Question about lFCD calculation with CPAC

[彌富泰佑]

Dear CPAC experts,

 Thank you very much for allowing me to participate in this wonderful mailing list. My name is Taisuke Yatomi, and I am a psychiatrist in Keio University School of Medicine in Japan. I am now conducting a certain resting-state fMRI study (rsfMRI) relating to PET data. In the study, I would like to extract local functional connectivity density (lFCD) from my rsfMRI data that is already equipped by your CPAC to calculate. Although I tried to do lFCD calculation using CPAC, the following errors were emerged after I conducted the script on my PC's terminal. Could you please share your extertise with me and explain how I could resolve these errors? Please note that rsfMRI data are several participants' ones after preprocessing and my configuration file is the attached one. I look forward to hearing from you.

My script on my PC(Mac Pro 2019)'s terminal: cpac -B /path/to/data/configs:/Users/t_yatomi/Downloads/fMRI_ongoingfiles group /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output --data_config_file /Users/t_yatomi/Downloads/CPAC_lfcd_ty.yml

Emerged errors:$ cpac -B /path/to/data/configs:/Users/t_yatomi/Downloads/fMRI_ongoingfiles group /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output --data_config_file /Users/t_yatomi/Downloads/CPAC_lfcd_ty.yml
Loading 🐳 Docker
Traceback (most recent call last):
  File "/Users/t_yatomi/opt/anaconda3/bin/cpac", line 8, in <module>
    sys.exit(run())
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/__main__.py", line 397, in run
    main(parsed)
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/__main__.py", line 326, in main
    Backends(**arg_vars).clarg(
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/backends/__init__.py", line 24, in Backends
    {
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/backends/docker.py", line 33, in __init__
    self._collect_config(**kwargs)
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/backends/docker.py", line 68, in _collect_config
    kwargs = self.collect_config_bindings(self.config, **kwargs)
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/site-packages/cpac/backends/platform.py", line 141, in collect_config_bindings
    os.makedirs(inner_binding, exist_ok=True)
  File "/Users/t_yatomi/opt/anaconda3/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/output'

Sincerely,
Taisuke

[Jon Clucas, MIS]

Hi Taisuke,

Unfortunately you’re hitting a known error in cpac (Python Package) on Mac OS ≥ 10.15.

Until we get that bug fixed (we’ll post back here once we do), you can run a C-PAC container directly through Docker or Apptainer / Singularity. All the C-PAC commands and configuration options are the same across the Python package and the container platforms; running directly just requires you as the user to specify some things that are abstracted away by the Python package, namely:

 

The different container platforms have different priorities; any blank cells in the table above are the default behavior of that platform. See these diagrams for a  conceptual visualization of directory bindings.

Please let us know if you have further difficulties, and watch this thread for a notification once we resolve the issue in the Python package.

[彌富泰佑]

Dear Jon,

 Thank you so much for your immediate response. I tried to conform to your advice, however, I got the following errors. I am afraid to bother you again, but would you mind teaching me some clues to get through these errors? (By the way, there is surely "CPAC_lfcd_ty.yml" in the "/Users/t_yatomi/Downloads/fMRI_ongoingfiles" directory on my PC) I hope to hear from you.

fMRI_ongoingfiles t_yatomi$ docker run -i --rm \
>         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func \
>         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output \
>         -v /tmp \
>         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles  \
>         fcpindi/c-pac:latest /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output participant --data_config_file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml --skip_bids_validator
#### Running C-PAC
Number of participants to run in parallel: 1
Output directory: /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output/output
Working directory: /tmp
Log directory: /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output/log
Remove working directory: True
Available memory: 1.0 (GB)
Available threads: 1
Number of threads for ANTs: 1
Error! Could not find config file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml

Traceback (most recent call last):

  File "/code/run.py", line 762, in <module>
    run_main()
  File "/code/run.py", line 656, in run_main
    args.aws_input_creds)
  File "/code/CPAC/utils/bids_utils.py", line 889, in load_cpac_data_config
    sub_list = load_yaml_config(data_config_file, aws_input_creds)
  File "/code/CPAC/utils/bids_utils.py", line 802, in load_yaml_config
    config_data = yaml.safe_load(open(config_filename, 'r'))
FileNotFoundError: [Errno 2] No such file or directory: '/Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml'

Best regards,
Taisuke

2023年1月13日(金) 0:53 Jon Clucas, MIS <jon.c...@childmind.org>:

--
You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/a8251635-b080-4cdd-b801-dc346170b725n%40googlegroups.com.

[Jon Clucas, MIS]

Please don't hesitate to reach out when you have troubles! Either we'll know the answer or you'll be informing us about a gap in documentation or a bug we're not aware of! (https://neurostars.org/tag/cpac is our preferred forum, but obviously we still check this one).

The current issue looks to be that the bind flags are missing the in-container paths. I suggest replacing

         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output \
         -v /tmp \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles  \

with

         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/func \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/output \
         -v /tmp:/tmp \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles:/Users/t_yatomi/Downloads/fMRI_ongoingfiles  \

so your in-container paths and local paths will be the same. And since /Users/t_yatomi/Downloads/fMRI_ongoingfiles wholly contains the first two bindings, you could actually just do

         -v /tmp:/tmp \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles:/Users/t_yatomi/Downloads/fMRI_ongoingfiles  \

instead of specifying the parent and two of its children.

You can name the in-container paths whatever you want so long as anything to the right of the image name (fcpindi/c-pac:latest) in the run command matches these bindings (including the contents of CPAC_lfcd_ty.yml). For example, you could do like

docker run -i --rm \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func:/input \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output:/output \
         -v /tmp:/tmp \
         -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml:/config/CPAC_lfcd_ty.yml  \
         fcpindi/c-pac:latest /input /output participant --data_config_file /config/CPAC_lfcd_ty.yml --skip_bids_validator

I personally find it easier to keep track of things by making the paths match, but using a particular in-container schema is handy for portability (e.g., if the runs will be moving amongst different environments or being run in different environments. This thread has some more bind mount examples.

Please let us know how it goes!

[彌富泰佑]

Dear Jon, 

  I appreciate your response, I comprehend the series of commands better. Though I conducted your recommended script, the following errors (2 versions) came up. I'm sorry to keep asking this, but could you please tell me how I tackle these matters? Thank you very much in advance. 

Errors emerged after conducting your recommended script at last: 

fMRI_ongoingfiles t_yatomi$ docker run -i --rm \

>          -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func:/input \
>          -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output:/output \
>          -v /tmp:/tmp \
>          -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml:/config/CPAC_lfcd_ty.yml  \
>          fcpindi/c-pac:latest /input /output participant --data_config_file /config/CPAC_lfcd_ty.yml --skip_bids_validator

#### Running C-PAC
Number of participants to run in parallel: 1

Output directory: /output/output
Working directory: /tmp
Log directory: /output/log

Remove working directory: True
Available memory: 1.0 (GB)
Available threads: 1
Number of threads for ANTs: 1

Traceback (most recent call last):
  File "/code/run.py", line 762, in <module>
    run_main()

  File "/code/run.py", line 665, in run_main
    if isinstance(sub.get('anat'), dict):
AttributeError: 'str' object has no attribute 'get'

Errors gotten after your recommended script at first:

fMRI_ongoingfiles t_yatomi$ docker run -i --rm \

> -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/func \
> -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/output \
> -v /tmp:/tmp \
> -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles:/Users/t_yatomi/Downloads/fMRI_ongoingfiles  \

> fcpindi/c-pac:latest /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output participant --data_config_file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml --skip_bids_validator
#### Running C-PAC
Number of participants to run in parallel: 1
Output directory: /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output/output
Working directory: /tmp
Log directory: /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output/log
Remove working directory: True
Available memory: 1.0 (GB)
Available threads: 1
Number of threads for ANTs: 1

Traceback (most recent call last):
  File "/code/run.py", line 762, in <module>
    run_main()

  File "/code/run.py", line 665, in run_main
    if isinstance(sub.get('anat'), dict):

AttributeError: 'str' object has no attribute 'get'

Kind regards, 

Taisuke

2023年1月13日(金) 3:56 Jon Clucas, MIS <jon.c...@childmind.org>:

--
You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/edf0578f-f1ea-433d-ae96-0af3e9212816n%40googlegroups.com.

[Jon Clucas, MIS]

🤔 Those look like syntax errors for the data config file. Are you comfortable sharing that file so I can take a look at the syntax?

[彌富泰佑]

Dear Jon, 

  Thank you very much for your prompt reply and precise advice. Could you please check the attached file? (Could it be the answer for your question "Are you comfortable sharing that file so I can take a look at the syntax?" ?)

Sincerely yours, 

Taisuke

2023年1月13日(金) 4:31 Jon Clucas, MIS <jon.c...@childmind.org>:

To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/22ce8efa-4005-4dd1-b37d-41424a958124n%40googlegroups.com.

[Jon Clucas, MIS]

• (Could it be the answer for your question "Are you comfortable sharing that file so I can take a look at the syntax?" ?)

Perfect, thank you!

• Could you please check the attached file?

Aha, yes. That's a pipeline configuration rather than a data configuration. If your input data are already in BIDS format, you should be able to just change

--data_config_file

to

--pipeline_file

(i.e., change

--data_config_file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml

to

--pipeline_file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml

)

 

[彌富泰佑]

Dear Jon, 

  I highly appreciate your quick look at my file. On my terminal, the following errors were seen after changing from --data_config_file to --pipeline_file. Actually, my rsfMRI data were not set as BIDS format, so I added "--skip_bids_validator" (Were current problems only that I did not change my rsfMRI data as BIDS format data? If so, I would try to mend them).  Would you mind telling me some advice to fix them? Thank you so much in advance. 

fMRI_ongoingfiles t_yatomi$ docker run -i --rm -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/func -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output:/Users/t_yatomi/Downloads/fMRI_ongoingfiles/output -v /tmp:/tmp -v /Users/t_yatomi/Downloads/fMRI_ongoingfiles:/Users/t_yatomi/Downloads/fMRI_ongoingfiles  fcpindi/c-pac:latest /Users/t_yatomi/Downloads/fMRI_ongoingfiles/func /Users/t_yatomi/Downloads/fMRI_ongoingfiles/output participant --pipeline_file /Users/t_yatomi/Downloads/fMRI_ongoingfiles/CPAC_lfcd_ty.yml --skip_bids_validator

Skipping bids-validator...

Traceback (most recent call last):

  File "/code/run.py", line 762, in <module>

    run_main()

  File "/code/run.py", line 478, in run_main

    c = Configuration(c)

  File "/code/CPAC/utils/configuration.py", line 129, in __init__

    config_map = schema(config_map)

  File "/code/CPAC/pipeline/schema.py", line 1071, in schema

    return latest_schema(_changes_1_8_0_to_1_8_1(config_dict))

  File "/usr/local/miniconda/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 272, in __call__

    return self._compiled([], data)

  File "/usr/local/miniconda/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict

    return base_validate(path, iteritems(data), out)

  File "/usr/local/miniconda/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 432, in validate_mapping

    raise er.MultipleInvalid(errors)

voluptuous.error.MultipleInvalid: extra keys not allowed @ data['pipeline_setup']['system_config']['observed_usage']

 (And also, I could not get data_config_file format since I did not fully understand the below command line, especially about what I should substitute for "$(id -u):$(id -g)". 

docker run -i --rm --user $(id -u):$(id -g) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config new_settings_template)

WIth kind regards, 

Taisuke

2023年1月13日(金) 4:51 Jon Clucas, MIS <jon.c...@childmind.org>:

--

You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/5b1caa33-0c1c-4eff-9ed5-a15310035ec6n%40googlegroups.com.

[Jon Clucas, MIS]

1. (And also, I could not get data_config_file format since I did not fully understand the below command line, especially about what I should substitute for "$(id -u):$(id -g)". 

docker run -i --rm --user $(id -u):$(id -g) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config new_settings_template)

No need to substitute. Just literally

docker run -i --rm --user $(id -u):$(id -g) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config new_settings_template

should work (

--user $(id -u):$(id -g)

tells Docker to run as your UID:GID so the files it creates will be owned by you). You can also just download the template from GitHub here, but you'll still need to fill it out and run the build command (

docker run -i --rm --user $(id -u) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config build /path/to/data_settings.yml

)

2. Actually, my rsfMRI data were not set as BIDS format, so I added "--skip_bids_validator" (Were current problems only that I did not change my rsfMRI data as BIDS format data? If so, I would try to mend them).  Would you mind telling me some advice to fix them?

`--skip_bids_validator` is for when you're confident your input data are in valid BIDS format and you don't want C-PAC to waste time and memory validating your file tree. You do need to create a data config if your input data are not in BIDS format. Otherwise C-PAC won't know how to parse the input file tree.

3. voluptuous.error.MultipleInvalid: extra keys not allowed @ data['pipeline_setup']['system_config']['observed_usage']

This one is surprising. That error is saying

pipeline_setup:

  system_config:

    observed_usage:

is an unknown key, but it's defined in the schema and in the default pipeline for the current version. Can you please run

docker run --rm -it --entrypoint python fcpindi/c-pac:latest -c "import CPAC; print(CPAC.__version__)"

and report back what it prints? I wonder if you've got an outdated image somehow.

[彌富泰佑]

Dear Jon,

 Ｉam now highly grateful for your precise and wonderful advice. The answer for your last question was "1.8.4" . (At first, the output was "1.8.3", but I found the value of version "1.8.4" in the GUI of CPAC pipeline file-making, and I got the value "1.8.4" after I tried the pipeline-configuration file.) Since I was going out for work and have to cease my research project during the daytime these days, but I tried all methods you mentioned in your previous emails at first after going back to my laboratory, and I found that CPAC ran partly, as you could see in the attached log (I also successfully arranged the resting-state fMRI data (preprocessed) and T1-weighted MRI data (preprocessed) in BIDS format). 

 But there seemed still something missed for completion of my pipeline that I would like to conduct and no results about LFCD data calculated were seen. To extract local functional connectivity density from my preprocessed resting-state fMRI data, I am now wondering if I should chage the type of configuration file or arrange the pipeline script in the configuration file "CPAC_LFCD_ty.yml". (If you could let me change my question, the question is whether it is possible for CPAC to skip the preprocessing (because I would like to use already preprocessed fMRI data) and obtain lFCD.) Could you please task a quick glance of my log or configuration file and suggest some solutions for my question? (In the "omission" part of my CPAC log, there were almost the same output logs of the subject 2~N(the number of subjects in my study) .) I look forward to hearing from you. 

With kind regards,
Taisuke

2023年1月13日(金) 5:45 Jon Clucas, MIS <jon.c...@childmind.org>:

--

You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/c3eb0b21-b076-469e-8c19-a92eaaba2019n%40googlegroups.com.

[彌富泰佑]

Dear Jon, 

　I apologize for the consecutive emails, but I found that I was able to proceed with several steps. Unfortunately, the attached following logs were the result of my script including the configuration file "CPAC_LFCD_ty5.yml". I tried to specify that the pipeline configuration file should work only for local functional connectivity (lFCD) extraction and "False" term was basically allocated manually (by me) for other modules in the default pipeline file. I am now wondering whether the errors were caused by the reason relating to the whole pipeline (for example, some appropriate templates and their location should have been specified or some time series extraction procedure should be done before lFCD extraction in the tool) or by the subtle reason that was shown in the last sentence of my terminal output "TypeError: argument of type 'NoneType' is not iterable" (that is, there were the only small script errors in my pipeline file that could be fixed relatively easily.)  Would you mind explaining what was the main error that should be fixed in this case (even if it could be just a brief advice)? I would appreciate it if you could tell me your expert advice. 

Sincerely yours, 

Taisuke

2023年1月16日(月) 1:53 彌富泰佑 <t.yato...@gmail.com>:

[Jon Clucas, MIS]

The first error

traits.trait_errors.TraitError: The 'schedule' trait of a FLIRTInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/usr/local/fsl/etc/flirtsch/bbr.sch' <class 'str'> was specified.

is a specific file-not-found. In both of these pipeline configurations, I see FSLDIR is changed from the default /usr/share/fsl/5.0 to /usr/local/fsl . It's fine if you want to use a different version / installation of FSL than the one that comes packaged with C-PAC, but to do so, you'll need to include it in your container bindings when you launch (add

-v /usr/local/fsl:/usr/local/fsl

to your command somewhere between `run` and the image name). The version of that not-found file packaged with C-PAC is at /usr/share/fsl/5.0/etc/flirtsch/bbr.sch ; you can use the packaged version by leaving it as the default /usr/share/fsl/5.0 or by setting it to the literal string FSLDIR .

The second error

  File "/code/CPAC/utils/monitoring/draw_gantt_chart.py", line 610, in _timing_timestamp
    k in {"start", "finish"} else v for k, v in node.items()}
  File "/code/CPAC/utils/monitoring/draw_gantt_chart.py", line 610, in <dictcomp>
    k in {"start", "finish"} else v for k, v in node.items()}

TypeError: argument of type 'NoneType' is not iterable

looks like no steps took any time in the almost-everything-off pipeline, so one of the resource-usage logs threw up. ( /outputs/log/crash-20230116-110902-root-check_for_s3-e7f50481-c1b0-46cf-b8be-89306a453836.txt might have more details or show another crash, but from that logfile that looks to me like that's all it is).

Looking at the code, I think network centrality depends on func_registration_to_template being on in the pipeline configuration. If you already have the outputs for that step in your bound output directory, C-PAC should consider that step complete and use the output from your output directory as a "cached" result file.

Please let us know if any of that ☝️ is unclear or if you have more questions.

[彌富泰佑]

Dear Jon, 

Thank you so much for your wonderful advice. I am sorry for my terribly delayed response. So far it has not worked out, but I will keep trying according to your previous advice. 

Best regards, 

Taisuke

2023年1月20日(金) 7:10 Jon Clucas, MIS <jon.c...@childmind.org>:

--

You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/1f7be968-50e1-4a8a-bf8d-1f11264a5dc1n%40googlegroups.com.