Re: [TVB] Need any help for the MRI preprocessing pipeline.

[WOODMAN Michael]

hi

The pipeline assumes you have already obtained an allocation on one of the supercomputers accessible from the collaboratory.  You can request access through Fenix https://jards-ebrains.fz-juelich.de

cheers,

Marmaduke

On 4 May 2023, at 10:20, 高政源 <joygaoz...@gmail.com> wrote:

I am a newcomer to trying to use the TVB and have found the pipeline Jupyter notebook to learn the pipeline step by step. When running the TVB PIPELINE after copying them from the pubic drive to my own collab, there are any mistakes as the picture show. Waiting for your help thanks.<1683188370482.png>

--
You received this message because you are subscribed to the Google Groups "TVB Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tvb-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tvb-users/a8e9b65c-6240-40f3-9686-7210cdd08ad5n%40googlegroups.com.
<1683188370482.png>

[Joey Gao]

Thanks for your help. I will see the details you provide to me. And I want to ask if I can use that 3 containers on my Linux service one by one to realize the pipeline. Or how can I pre-process my MRI data offline?

[WOODMAN Michael]

hi

You can download the container images to your own computer and process the data, yes.  For data protection purposes, this may be easier than using Ebrains or related services, for a handful of subjects.

cheers,

Marmaduke

To view this discussion on the web visit https://groups.google.com/d/msgid/tvb-users/61bf6b53-3b3c-4e5b-86e6-f541054375adn%40googlegroups.com.

[Joey Gao]

Hello, sorry to bother you. I am not sure how to use these three containers on my own computer. I have these questions, can you help me?
1. I have pulled the containers on my computer, can I preprocess that only with mode 3 as the tvb_pipeline_v1.1 step10? From my understanding steps 1-9 are the communication steps with the cloud. Right?   

2. I have installed the slurm but when I run the mode 1, there are errors about "unable to locate a modulefile for daint-mc and sarus", how can I deal with this?

3. Is there any quick tutorial to guide the newcomers to preprocess the MRI data? 

• I didn't find a guide to a good offline process

Waiting for your reply. Thank you very much.

[Michael Schirner]

Hi Joey,

1. Yes, these steps are for communication between EBRAINS and Fenix HPC sites. To process locally, you can follow the documented procedures of the underlying BIDS Apps MRtrix3_connectome and fmriprep: 

https://hub.docker.com/r/bids/mrtrix3_connectome/

https://hub.docker.com/r/nipreps/fmriprep/

2. Slurm is a job queue manager for supercomputers. If you process locally, you may not need Slurm. Likewise, you may not need the orchestrator script, as they are mostly intended to manage the workflow between a remote user and the Fenix HPC sites. It can however serve as inspiration for how to combine Encryption and DataLad with this workflow. To process locally, it is probably most efficient to run MRtrix3_connectome and fmriprep containers on your data and then run the converter (https://hub.docker.com/r/thevirtualbrain/tvb-pipeline-converter) on the outputs, by directly calling the container via "docker run" commands.

3.  I suppose there is no "quick" tutorial, as these pipelines are quite complex and there are many pitfalls. The documentations of MRtrix3 and fmriprep and these publications may help:

https://pubmed.ncbi.nlm.nih.gov/23668970/

https://pubmed.ncbi.nlm.nih.gov/25837600/

I found this tutorial helpful for MRtrix propressing:

https://osf.io/fkyht/

Hope this helps!

Best wishes,

Micha

[Joey Gao]

Thanks for your help. 

Actually, I am also trying the pipeline of the paper " An automated pipeline for constructing personalized virtual brains from multimodal neuroimaging data " you provide to me. I'm not sure if you are familiar with that pipeline. I also met some questions.

The input of this pipeline is the dicom format without any demo data. So I download any dicom data from the ADNI. But when it runs, there's an error showing "ERROR! Not enough distinct directions in your dwMRI data! Sorry". I am not sure if this error is a problem with the ordinary data. If that what kind of data I can use with this pipeline? 

Waiting for your reply.

Also, I am trying my best with the dockers. As I know, the input of this pipeline is the BIDS format, which will meet little problem with the data format, I think. 

Thanks again.

[Michael Schirner]

I would suggest to stick with the fmriprep and MRtrix3_connectome containers, as they are actively maintained and updated. The error indicates that the dwMRI data has not enough diffusion gradient directions ("b-vectors"). For dwMRI beginners I would advise to manually follow each step of the documentation and tutorial for one or two subjects and to closely inspect the inputs and outputs, in order to get a deeper understanding of the data and what each processing step does. fMRI processing with fmriprep usually works more robustly and here chances are higher that it may successfully execute without errors.

[Joey Gao]

Thanks for your suggestion. I will try as your reply.

[Joey Gao]

As your suggestion. I have run the fmriprep and MRtrix3_connectome containers successfully. For the Mrtix4, there is an MRtrix3_connectome-preproc folder. And, there are .bidsignore, dataset_description.json, descxxx.tsv, logs, sourcedata, sub-XXX, and sub-XXX.html.  Am I on a right way? 

If yes,  I think I need to use the tvb-converter to convert the results to the TVB format. But I don't find any order code to convert. If I read the code on GitHub? or there is a command to convert it in one line?

Thanks for your time. Waiting for your reply.

[Michael Schirner]

Yes, I think you are on the right way: these files/folders indicate that MRtrix3_connectome, respectively fmriprep, are generating outputs. Please see the container command in the call script pipeline.sh.

[Joey Gao]

Thanks, I tried this way. And I believe that's the right way too. 

And there's any new question.

1. I tried this pipeline with the demo dataset with the sub-CON03.  And for this data, there's a /ses-postop/ path between the /sub-CON03/ and the /anat&dwi&func/. And I noticed that the path set in the converter doesn't contain the /ses-postop/ like 

But my path are like

 

So this scipt are not suit do process for all of the data. I need to modify it to suit for my case. Right?

2. Before you share me the pipeline.sh script. I used the docker run command to generate the MRtrix3_connectome-preproc&participant two results in the mrtrix_output folder.

 

And I noticed using the short workaround way will save the recon-all results and It seems I must run it in this wany or I can't realize the convertion, right?.

 But when I use the short workaround way. There’s a problem. Here’s my code and the error information.

 

Waiting for your reply. Thanks very much, again.

[Joey Gao]

Here's a supplement.

If I use the pipeline directly with the path changing. Here's the same error. It will jump to fmri processing directly.

[Michael Schirner]

Hi Joey,

1. Right.
2. Right, the recon-all output is needed for the subsequent conversion. I think the problem results from a change in the MRtrix3_connectome call command structure. It was changed from a one-stage call to a two-stage call structure, with "preproc" and "participant" stages, as you mentioned. To mitigate I see two strategies: First, you could replace the call

docker run ... bids/mrtrix3_connectome with either

docker run ... thevirtualbrain/tvb-pipeline-sc or with
docker run ... bids/mrtrix3_connectome:0.5.0

In the first variant you would use a snapshot of the older bids/mrtrix3_connectome container that we archived for long-term availability. Alternatively, you could specify the tag of the older container as in the second variant.

Alternatively, you could also try changing the pipeline script to work with the newer version of the bids/mrtrix3_connectome container. Here I would recommend to build up the command step-by-step by first running it with default settings as documented. The relatively complex command as needed here, may likely not be necessary anymore with the newer version.

Hope this helps!

Best,
Micha

[Joey Gao]

Sorry to interrupt you.  I want to make sure one point about the command of the mrtrix processing.  I am not sure if I need to use docker to run the mrtrix container directly or just modify the docker container name in the short workaround of the pipeline.sh.
Or if it is necessary for the short workaround way to run the mrtrix processing?  

Thanks, Michael.

[Joey Gao]

It seems that only changing the docker name in pipeline.sh can't solve the problem. That's my pipeline.sh and the same error. I am so confused.

[Michael Schirner]

Sorry about the confusion, I fully understand it. Unfortunately, I'm not able to offer a pipeline that "just works on button press" for every dataset and/or continues to do so over time. The pipeline has dozens of interacting third-party components and to make things worse the interfaces of these components change over time. With containerization and modularization we try to mitigate this issue, but still it's not easy to solve this problem of breaking changes of software interfaces and non-standardization of brain data. I see unfortunately no other way than step-by-step debugging such errors.

I can see two issues here:

1. "AttributeError: 'NoneType' object has no attribute 'rfind'" is a problem when MRtrix3_connectome container tries to parse the input command line. We could now try to find out what is wrong here, but my suggestion would be to rather built the command line new from scratch with the latest bids/MRtrix3_connectome container and its new call structure. There were also updates on the scientific MRtrix3 workflow side -- I think they announced critical errors in the tractography algorithm and continue to update their code base with fixes and improvements, so I would suggest that track the latest MRtrix development branch and debug all conflicts. I may be able to help here after you successfully used the latest MRtrix3 version to perform tractography and produce the raw connectomes.

2. "mkdir: cannot create ..." and "cp: cannot stat '/freesurfer..." are steps in the pipeline script here that come after the MRtrix step and fail because the MRtrix step fails, so here I would suggest to also first solve the MRtrix problem.

Sorry that I cannot be of more help here. 

Another idea: there are also the pipelines

https://github.com/McIntosh-Lab/tvb-ukbb

and

https://github.com/the-virtual-brain/tvb-recon

Maybe they are more practical for your use case?

[Joey Gao]

Whatever, thanks for your suggestion and patience very much. 

[Joey Gao]

Hello Michael,

Sorry to interrupt you again. For the question above, it seems that if I use the docker run command with the --output_verbosity 4 combining without the --entrypoint args, the scratch folder will save in the output file. So I will get recon-all results with the mrtrix processing. It is running now, I will try to convert again (I think all the path needed in the convertion is specific).

But I also have one question to make sure. As I know, we have 2 workpaths named "tmp" in fMRI and TVB-output. And the fMRI tmp is needed to provide the conversion processing. Are there any files saved in the fmri tmp file? I remember the folder is empty without any results in my mind. Is it right?

[Joey Gao]

Hello Michael,

It seems that I have run the tvb_converter_pipeline.sh successfully. But there's one question existing.

In the tvb_converter_pipeline.sh, there's a "flirt" command to make the "anat/sub-${participant_label}_parc-${parcellation}_indices.nii.gz" as the input. But due to the change in the two stages processing of mrtrix, now the out put is as that.

I think that's a name-changing of the software update. If right, I will change the name as your pipeline to try, thanks very much.

Best wishes.

It seems near to finish.

Thanks~

[Joey Gao]

Hello, Michael,

I found that even the flirt command is not finished working, the pipeline still going working. and now I get a new problem, I am not sure is the problem of flirt processing or some part I made the mistake. Waiting for your reply, thanks very much.

[Michael Schirner]

Hi Joey,

the error "Surface inner skull is not completely inside surface outer skull" is produced by FreeSurfer. This problem occurs sometimes in the recon-all run. Google has some suggestions on how to deal with it, I think it can be mitigated by changing the settings of FreeSurfer's watershed algorithm.

[Joey Gao]

Thanks for your reply about the " Surface inner skull is not completely inside surface outer skull " error. I will try to find a way to fix it, especially using the docker. I am not very familiar with docker.

And how about another question about the flirt command? Do you have any suggestions for that?

[Michael Schirner]

Regarding Docker, as an alternative, FreeSurfer can also be installed locally and recon-all can be run without container. Regarding Flirt, I didn't catch the question?

[Joey Gao]

Hello Michael, I mean this one.

In the tvb_converter_pipeline.sh, there's a "flirt" command to make the "anat/sub-${participant_label}_parc-${parcellation}_indices.nii.gz" as the input. But maybe due to the change in the two stages processing of Mrtrix3, now the output of the Mrtrix3 is as that.

I'm unsure if that's a name-changing of the software update. 

So I don't know which one I need to change the name to run the flirt command. Or it is necessary for the flirt command?

Thanks

[Michael Schirner]

Hi Joey, indeed, this looks like a change of the naming convention. MRtrix, fmriprep, and other apps use the BIDS standard as data format. Probably the suffix "dseg" to indicate parcellation images was only introduced after the previous version of MRtrix was written and was now updated in the new version. As a practical consequence: yes, you would need to adapt the filename accordingly.

[Joey Gao]

Thanks for your reply. I will try.

[Joey Gao]

Hello, Michael.
Here's a new question. Can you help me?
The test data is like this in dwi file, And it works.

But when I use my real data. There's an error making the script stopped.
The data list in dwi and the error information are as below.


If we need the acp-AP and acp-PA data at the same time？ Can I  transfer my file to the right data structure. If yes, can you tell me how to do that?

[Schirner, Michael]

Hi Joey,

yes, you need the acp-AP and acp-PA data at the same time. The exact steps for converting your data to the required format depend on your specific data. For example, if your file `sub-RC4101_dwi.nii.gz` contains scans acquired with both phase encoding directions (AP and PA), then you could split them like in the first screenshot using the `acq-` filename entity (the MRtrix commands mrinfo and mrconvert are helpful for that purpose). Please also note the requirements regarding metadata in sidecar JSON files for MRtrix to correctly interpret the different AP/PA files, as specified in the MRtrix3_connectome documentation. Please also see the main MRtrix documentation for handling of phase encoding schemes.

Best,

Micha

---

From: tvb-...@googlegroups.com <tvb-...@googlegroups.com> on behalf of Joey Gao <joygaoz...@gmail.com>
Sent: Saturday, June 10, 2023 12:48:32 PM
To: TVB Users
Subject: [ext] Re: [RESEAUX SOCIAUX] [TVB] Need any help for the MRI preprocessing pipeline.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/tvb-users/0d3efa9b-a514-455b-867c-13486bcc31dfn%40googlegroups.com.