fmriprep + bids derivatives + directory layout

[Christina Roßmanith]

Hi,

I'm quite new to the BIDS subject and have arranged my data so that the validator is happy. Now I'm using fmriprep and had to choose an output directory. I've chosen ./fmriprep parallel to ./BIDS which is the toplevel directory for my BIDS data set.

Seeing that fmriprep produces derivative data, maybe it would be better to move the fmriprep folder into ./BIDS/derivatives/fmriprep?

So what are the experts recommendations regarding the directory layout?

Regards,
Christina

[Earl, Eric (NIH/NIMH) [C]]

Christina,

 

You are quite right! The fmriprep/ processed derivatives subdirectory should go in the derivatives/ subdirectory from your main dataset (./BIDS/derivatives/fmriprep-<version>). For more on that, read up here:

 

https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#storage-of-derived-datasets

 

Good luck and happy deriving!

 

Warm regards,

 

Eric Earl [C]

Scientist

 

eric...@nih.gov

(503) 752-8549

 

NIMH IRP Data Science & Sharing Team

https://cmn.nimh.nih.gov/dsst

Google Scholar Profile

https://scholar.google.com/citations?user=8IwbaOcAAAAJ

Book a meeting

https://calendly.com/ericearl

--
We are all colleagues working together to shape brain imaging for tomorrow, please be respectful, gracious, and patient with your fellow group members.
---
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/f7a0f482-57bd-4938-bd78-f97be417fee5n%40googlegroups.com.

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.

 

[Chris Markiewicz]

Hi Eric and Christina,

Note that derivatives as a subdataset is one of two approaches listed in that section. Derivatives can be standalone datasets, and it really depends on how you want to organize things.

As an alternative example, as we process OpenNeuro datasets, we include the original dataset as a sub-dataset of the derivative. See https://github.com/OpenNeuroDerivatives/ds000001-fmriprep, where we include the raw dataset as sourcedata/raw. Because the datasets are DataLad datasets (i.e., git repositories), the derivative contains a link to the exact version of the raw dataset that it processed on, a workflow that follows the YODA principles (https://handbook.datalad.org/en/latest/basics/101-127-yoda.html).

This may or may not be how you want to organize your derivatives, but there's not just one way to do it.

Best,

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/BY3PR09MB72848735238CA14E25B1F8B3EC8F9%40BY3PR09MB7284.namprd09.prod.outlook.com.

[Oostenveld, R. (Robert)]

Hi Christina

Both are fine and consistent with the documentation at https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#source-vs-raw-vs-derived-data

A consideration that I use for myself is that, once I have preprocessed the data, I don’t need to keep the previous step on the (often space-limited) storage system. In our case the fast network storage is attached to the compute cluster and is quota limited. So my approach is like this

/home/robert/projectABC/

├── sourcedata

├── bids_raw (depends on sourcedata)

├── bids_derivative1 (depends on raw)

└── bids_derivative2 (depends on derivative1)

where each of them can be deleted (aka archived to slow and low-cost storage) once the next step is complete and validated. Documenting in the dataset_description what processing was done and where the previous data can be found is important; that is simpler if you use the nested organization with subdirs and subsubdirs in bids_raw. 

best regards

Robert

 

[Chris Markiewicz]

Hi Christina,

The BIDS validator does not yet fully support BIDS derivatives. This will be supported when we transition to the (in-progress) schema-based validator. You can read about using the schema-based validator here: https://github.com/bids-standard/bids-validator/tree/master/bids-validator/src#intro

Best,

Chris

On Tue, Apr 4, 2023 at 5:35 AM Christina Roßmanith <christina....@medma.uni-heidelberg.de> wrote:

I have moved fMRIPrep to BIDS/derivatives and thought I could try to bids-validate the fMRIPrep output. But this fails with lots of "file name not bids conform" errors. I'd like to try bids/spm for fMRI analysis but I fear if the validator fails spm_BIDS fails on BIDS/derivatives/fMRIPrep as well. I've used the validator in the browser which should be quite a recent version?!? The directory BIDS is validated with a few warnings only.

Any hints?

Regards,
Christina

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/c7805440-4d8f-416b-9fe4-b1836eec194dn%40googlegroups.com.

[Remi Gau]

Regarding fmriprep output and the validator, see this recent post on neurostars: https://neurostars.org/t/fmriprep-output-bids-validator/25446

I have not checked spm_BIDS may indeed very well fail on derivatives as I think it is mostly designed to handle raw BIDS but for using bids derivatives from fmriprep, you should be able to use either pybids and bids-matlab to "ingest" them for your analysis:

• see doc for pybids: https://bids-standard.github.io/pybids/examples/pybids_tutorial.html#loading-derivatives
• see doc for bids-matlab: https://github.com/bids-standard/bids-matlab/blob/dev/demos/notebooks/BIDS_Matlab_02_derivatives.m

Hope this helps.

Rémi

On 03/04/2023 14:49, Christina Roßmanith wrote:

I have moved fMRIPrep to BIDS/derivatives and thought I could try to bids-validate the fMRIPrep output. But this fails with lots of "file name not bids conform" errors. I'd like to try bids/spm for fMRI analysis but I fear if the validator fails spm_BIDS fails on BIDS/derivatives/fMRIPrep as well. I've used the validator in the browser which should be quite a recent version?!? The directory BIDS is validated with a few warnings only.

Any hints?

Regards,
Christina

Chris Markiewicz schrieb am Freitag, 31. März 2023 um 17:41:47 UTC+2:

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/c7805440-4d8f-416b-9fe4-b1836eec194dn%40googlegroups.com.

--

Rémi Gau