how to run cpac after configure the data_setttings.yml and data_config.yml?

[Xin Niu]

Dear cpac experts,

I have successfully set up the data_setting.yml and data_config.yml file with the following docker commands:

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataConfigPath:/scratch \
-w="/scratch" \
fcpindi/c-pac:latest 'xx' 'xx' cli -- \
utils data_config new_settings_template

and:

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \
-v $sataSettingPath:/scratch \
-w"/scratch" \
fcpindi/c-pac:latest $dataPath $outputPath cli -- utils \
data_config build $dataSettingPath/data_settings.yml

But I have no clue what the next steps should be.

I tried run:

cpac run out02_adni_fmri_t1/ cpac_output_dir/ participant

But the system cannot find cpac:

Command 'cpac' not found, did you mean:

  command 'cpan' from deb perl (5.30.0-9ubuntu0.2)
  command 'crac' from deb crac (2.5.2+dfsg-2build2)
  command 'pac' from deb lpr (1:2008.05.17.3)
  command 'cpack' from deb cmake (3.16.3-1ubuntu1)

Try: sudo apt install <deb name>

This happens after I install cpac with pip:

xin@xin-Precision-3650-Tower:~/Downloads/BrainImaging_UNC$ pip install cpac
Requirement already satisfied: cpac in /home/xin/.local/lib/python3.8/site-packages (0.4.0)
Requirement already satisfied: tornado in /home/xin/.local/lib/python3.8/site-packages (from cpac) (6.1)
Requirement already satisfied: websocket-client in /home/xin/.local/lib/python3.8/site-packages (from cpac) (1.3.1)
Requirement already satisfied: docker-pycreds in /home/xin/.local/lib/python3.8/site-packages (from cpac) (0.4.0)
Requirement already satisfied: docker in /home/xin/.local/lib/python3.8/site-packages (from cpac) (5.0.3)
Requirement already satisfied: spython>=0.0.81 in /home/xin/.local/lib/python3.8/site-packages (from cpac) (0.1.18)
Requirement already satisfied: pandas>=0.23.4 in /home/xin/.local/lib/python3.8/site-packages (from cpac) (1.4.1)
Requirement already satisfied: tabulate>=0.8.6 in /home/xin/.local/lib/python3.8/site-packages (from cpac) (0.8.9)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from cpac) (5.3.1)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from docker-pycreds->cpac) (1.14.0)
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker->cpac) (2.22.0)
Requirement already satisfied: semver>=2.8.1 in /home/xin/.local/lib/python3.8/site-packages (from spython>=0.0.81->cpac) (2.13.0)
Requirement already satisfied: pytz>=2020.1 in /home/xin/.local/lib/python3.8/site-packages (from pandas>=0.23.4->cpac) (2021.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /home/xin/.local/lib/python3.8/site-packages (from pandas>=0.23.4->cpac) (2.8.2)
Requirement already satisfied: numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10" in /home/xin/.local/lib/python3.8/site-packages (from pandas>=0.23.4->cpac) (1.22.3)

I am wondering if there is a way to run the analysis (for all subjects) with docker?

Thanks for reading this email. Any help is appreciated!

[Jon Clucas, MIS]

Hi,

Thanks for reaching out for help.

We have two similarly-named tools:

1. C-PAC (the Configurable Pipeline for the Analysis of Connectomes) which runs in a Docker or Singularity container
2. cpac (Python package) which is a tool intended to facilitate running C-PAC by abstracting away the container interactions (e.g., so users don't have to bind all the necessary paths themselves)

To run C-PAC directly with Docker, check out C-PAC latest documentation » Run on Docker. For that option, your command would look something like

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \

fcpindi/c-pac:latest $dataPath $outputPath run participant --data_config_file $dataSettingPath/data_config_${SUBJECTLISTNAME}.yml

(You probably don't need -v $dataSettingPath:/scratch -w "/scratch"; if you want to save the working directory, you can just use the --save_working_dir flag)

To run C-PAC with cpac, `pip install cpac` is the right way to install. That install log looks incomplete but that it was installing okay. If you have pip version conflicts, it's a good idea to install cpac in its own Python environment (e.g., using pyenv or conda or venv or some other manager). For that option, your command would look something like

cpac run $dataPath $outputPath participant --data_config_file $dataSettingPath/data_config_${SUBJECTLISTNAME}.yml

Please let us know if you're following along in the user docs and get stuck somewhere.

Thanks,

Jon

[Xin Niu]

Hi Jon,

Thanks for your quick reply!

I tried:

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \
fcpindi/c-pac:latest $dataPath $outputPath run participant \

--data_config_file $dataSettingPath/data_config_data_config_adni.yml

But it there is an error:

run.py: error: argument analysis_level: invalid choice: 'run' (choose from 'participant', 'group', 'test_config', 'cli')

Shall I rerun the previous steps (also omitting the option: -v $dataSettingPath:/scratch -w "/scratch")? 

My understanding is -v $dataSettingPath:/scratch -w "/scratch" in the previous option save the *.yml files in the working directory (dataSettingPath), so it doesn't have to be consistent with the following command. Is that correct?

And also if I want to save the working directory:

shall I add:

--save_working_dir flag

or:

--save_working_dir True

or:

--save_working_dir $outputPath

Thank you!

Best,

Xin

[Jon Clucas, MIS]

I tried:

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \
fcpindi/c-pac:latest $dataPath $outputPath run participant \

--data_config_file $dataSettingPath/data_config_data_config_adni.yml

But it there is an error:

run.py: error: argument analysis_level: invalid choice: 'run' (choose from 'participant', 'group', 'test_config', 'cli')

Sorry, that's my mistake. I'm not sure how that extra 'run' made its way in there, but the command should be like

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \

fcpindi/c-pac:latest $dataPath $outputPath participant \
--data_config_file $dataSettingPath/data_config_data_config_adni.yml

(without the 'run' between '$outputPath' and 'participant' ― 'bids_dir', 'output_dir' and 'level_of_analysis' are positional arguments, so having 'run' in between the second and third is causing the error).

Shall I rerun the previous steps (also omitting the option: -v $dataSettingPath:/scratch -w "/scratch")?

If you have a data config, you shouldn't need to regenerate unless your input data structure has changed.

My understanding is -v $dataSettingPath:/scratch -w "/scratch" in the previous option save the *.yml files in the working directory (dataSettingPath), so it doesn't have to be consistent with the following command. Is that correct?

The key outputSubjectListLocation in data_settings.yml specifies where in-container to save your data config file when you run utils data_config build, so if that path is set to /scratch, you're absolutely correct. Otherwise, you'd just need to bind whatever path you're using in place of /scratch. The reason I said you probably don't need -v $dataSettingPath:/scratch -w "/scratch" is that C-PAC automatically creates a working directory called 'working' at the same level as 'log' and 'output' in your output directory if you use the --save_working_dir flag without any parameters, and if you don't use --save_working_dir, the working directory is deleted at the end of the run, so it doesn't matter where those ephemeral working data are written.

And also if I want to save the working directory:

shall I add:

--save_working_dir flag

or:

--save_working_dir True

or:

--save_working_dir $outputPath

The simplest thing is just --save_working_dir, like

docker run -i --rm --user $(id -u):$(id -g) \
-v $dataPath:$dataPath \
-v $outputPath:$outputPath \
-v $dataSettingPath:$dataSettingPath \

fcpindi/c-pac:latest $dataPath $outputPath participant \
--data_config_file $dataSettingPath/data_config_data_config_adni.yml \

--save_working_dir

which will save your working directory in $outputPath/working. If you want it somewhere else, you can provide a path after the flag, but make sure you also bind that path (unless it's an S3 bucket)