Error on data_setting.yml
이지원
Jon Clucas, MIS
이지원
--
You received this message because you are subscribed to a topic in the Google Groups "cpax_forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cpax_forum/0dzk9MDqK_E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cpax_forum+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/9b740322-906c-4093-97c3-cd003987ad77n%40googlegroups.com.
Jon Clucas, MIS
[--platform {docker,singularity}] [--image IMAGE] [--tag TAG]
[--working_dir PATH] [-v] [-vv]
{run,group,utils,pull,upgrade,crash} ...
cpac: a Python package that simplifies using C-PAC <http://fcp-indi.github.io> containerized images.
This commandline interface package is designed to minimize repetition.
As such, nearly all arguments are optional.
When launching a container, this package will try to bind any paths mentioned in
• the command
• the data configuration
directories to bind with a different path in
the container than the real path of the directory.
One or more pairs in the format:
real_path:container_path
(eg, /home/C-PAC/run5/outputs:/outputs).
Use absolute paths for both paths.
This flag can take multiple arguments so cannot be
the final argument before the command argument (i.e.,
run or any other command that does not start with - or --)
----------------- --------- ------
/home/djk /home/djk rw
/home/djk /tmp rw
/home/djk/outputs /output rw
/home/dkk/log /crash rw
and be able to write to that path, but that path won't exist if the container is launched from a working directory outside of /media[--data_config_file DATA_CONFIG_FILE] [--preconfig PRECONFIG]
[--aws_input_creds AWS_INPUT_CREDS]
[--aws_output_creds AWS_OUTPUT_CREDS] [--n_cpus N_CPUS]
[--mem_mb MEM_MB] [--mem_gb MEM_GB]
[--num_ants_threads NUM_ANTS_THREADS]
[--save_working_dir [SAVE_WORKING_DIR]] [--disable_file_logging]
[--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
[--participant_ndx PARTICIPANT_NDX] [--T1w_label T1W_LABEL]
[--bold_label BOLD_LABEL [BOLD_LABEL ...]] [-v]
[--bids_validator_config BIDS_VALIDATOR_CONFIG]
[--skip_bids_validator] [--anat_only] [--tracking_opt-out]
[--monitoring]
bids_dir output_dir {participant,group,test_config,cli}
bids_dir The directory with the input dataset formatted
according to the BIDS standard. Use the format
s3://bucket/path/to/bidsdir to read data directly from
an S3 bucket. This may require AWS S3 credentials
specified via the --aws_input_creds option.
output_dir The directory where the output files should be stored.
If you are running group level analysis this folder
should be prepopulated with the results of the
participant level analysis. Use the format
s3://bucket/path/to/bidsdir to write data directly to
an S3 bucket. This may require AWS S3 credentials
specified via the --aws_output_creds option.
{participant,group,test_config,cli}
Level of the analysis that will be performed. Multiple
participant level analyses can be run independently
(in parallel) using the same output_dir. test_config
will run through the entire configuration process but
will not execute the pipeline.
- Bind /media/12T/ABIDE/any_dir to a read-only directory in your container at /media/12T/ABIDE/any_dir . Because none of that path exists yet in the container, the whole tree (/media/12T/ABIDE, /media/12T, and /media) is read-only.
- Bind /media/12T/ABIDE/case1_s3_default_output to a read-write directory in your container at /media/12T/ABIDE/case1_s3_default_output . Because /media/12T/ABIDE in the container is read-only, this bind fails
- Because Docker failed and the platform wasn't specified, cpac tries to do the same thing with Singularity instead of Docker ("By default, cpac (the wrapper) will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised."). Because this crash is not platform-specific, Singularity behaves the same as docker (binds read-only directory, fails to bind read-write subdirectory)
- Bind /media/12T/ABIDE to /media/12T/ABIDE and to /tmp inside the container (both paths are read-write)
- Bind /media/12T/ABIDE/any_dir to a read-only directory in your container at /media/12T/ABIDE/any_dir . Because the parent directory already exists, the parent tree keeps its existing permissions.
- Bind /media/12T/ABIDE/case1_s3_default_output to a read-write directory in your container at /media/12T/ABIDE/case1_s3_default_output . Because /media/12T/ABIDE in the container is read-write, this bind succeeds
- [Continue processing]
이지원
--
You received this message because you are subscribed to the Google Groups "cpax_forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpax_forum+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cpax_forum/cbee45c8-3dd9-4b05-b13f-d33232f72dd7n%40googlegroups.com.
이지원
Jon Clucas, MIS
1. “Containers isolate software from its environment;” the containers are created somewhere not intended for human use, that location depends on your container platform configuration, and you typically shouldn’t need to know or care where the container is created (e.g., if the container location is on a partition with inadequate space).
The root tree in the current latest C-PAC container image looks like
.
├── ants_template
├── bin
├── boot
├── code
├── cpac_resources
├── cpac_templates
├── dev
├── etc
├── home
├── lib
├── lib64
├── media
├── mnt
├── ndmg_atlases
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── @update.afni.binaries
├── usr
└── var
and /media is empty. cpac (the command line interface) tries to bind all the local paths you’ll need in your container, mostly at matching names (the path names don’t have to match). When a local path is bound to a path in a container, any files or subdirectories in that local path are also accessible at that path in the container and vice versa. Any changes locally are reflected in the container, and any changes in the container are reflected locally but only at bound paths. When you run cpac, your terminal will print a little table at the top that tells you which paths locally are bound to which paths in the container and what the access mode is for the binding.
For any unspecified paths that cpac thinks C-PAC might need, cpac assumes the current working directory, so in most cases the current working directory is bound at runtime by cpac in a C-PAC container to a matching path and to C-PAC's working directory.
2. The in-container working directory is defined in the pipeline configuration under pipeline_setup.working_directory.path. For the default configuration, this path is set to /tmp (and in most cases this path shouldn’t matter from a user's perspective).
3. According to the BIDS-app specification, output_dir (positional argument #2) “is the only directory the pipeline should write to,” so this directory has to be writable.
4. By default, bids_dir (positional argument #1) is read-only, output_dir (positional argument #2) and the logging and working directories are read-write, and any custom bindings that aren’t specified read-only are defaulted to read-write.
5. Yes.
Running from the home directory with /media/12T/ABIDE/any_dir as the read-only bids_dir makes /media/12T and its descendant directories read-only, preventing the output directory, a child of /media/12T, from being created. If the output directory (/media/12T/ABIDE/case1_s3_default_output) already existed, that run may have succeeded. I’m not sure if a child of a local directory bound read-only can be bound read-write ― the failure you reported was the attempt to create a read-write child in a read-only directory.
Running from /media/12T/ABIDE that directory is bound read-write to paths in the container that aren’t otherwise specified (itself as the current working directory and /tmp as the C-PAC working directory), so cpac is able to create a child directory within that path.
We (the development team) could resequence these bindings so that read-only directories are bound last, but I'm concerned that would make accidentally writing to a directory that should be read-only too easy.
6.
cpac utils data_config build binds your current working directory to
a matching path and to /output inside the container.
(The table that prints to terminal will say
Loading 🐳 Docker
Loading 🐳 fcpindi/c-pac:latest with these directory bindings:
local Docker mode
---------------- ---------------- ------
/media/12T/ABIDE /media/12T/ABIDE rw
/media/12T/ABIDE /output rw
Logging messages will refer to the Docker paths.
)
In your container your root file tree will include
/
├──
media
│
└── 12T
│ └── ABIDE
│ └── «
files and directories in local /media/12T/ABIDE »
└──
output
└── «
files and directories in local /media/12T/ABIDE »
And any changes in your local /media/12T/ABIDE or your in-container /media/12T/ABIDE or your in-container /output will be reflected in all three locations. Any other paths in your local /media/12T or local /media will be unknown in your container environment.
Hopefully these answers help. Please ask more questions if you need more clarification.
Thanks,
Jon