Efficient download & decompression of HCP rfMRI datasets

[yi yang]

Hi everyone,

I am currently working on analyzing brain connectivity using rfMRI data. After downloading the preprocessed rfMRI data of a certain subject, I realized that I only need four specific files (REST1_LR, REST1_RL, REST2_LR, REST2_RL):

MNINonLinear\Results\rfMRI_RESTx_XX\rfMRI_RESTx_XX_Atlas_MSMAll_hp2000_clean_rclean_tclean.dtseries.nii

However, it seems that on BALSA, the data can only be downloaded as a complete package (e.g., the entire xxx_Rest3TRecommended directory). I have been using IBM Aspera Connect on Windows, which gives me a transfer speed of over 100 Mbps, but due to the huge size of the full dataset, the download could still take up to two weeks.

In addition, I also noticed that decompressing the downloaded package takes even longer than expected, which makes the overall process quite time-consuming.

Therefore, I would like to ask:

1. Is there a way to download only specific files or subdirectories, instead of the full package?
2. Are there any recommended tips to use IBM Aspera Connect more efficiently to reduce the total download time?
3. Are there any suggestions to speed up or simplify the decompression of these very large packages?

I also want to take a moment to thank the HCP team for making this valuable dataset available, and the community members who often share their experiences and solutions here. Your work and support are truly appreciated!

Best wishes,

Yi

[Glasser, Matthew]

Not currently, but it is planned.

 

Matt.

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/412eace4-1908-42b6-a851-9133c1d78902n%40humanconnectome.org.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Tim Coalson]

Most zip utilities have a way to specify extraction of only a subset of files, which may save a bit of time on the extraction step.  If the bottleneck is cpu rather than storage speed, you could see about having more than one subject decompressing at the same time (just try not to exceed the number of cores in the system, as that would start to make things less efficient).

Tim

--