Downloading a particular file

[najme soleimani]

Dear HCP experts,

I wonder if I could download a specific file within a package for a group of subject without downloading the whole package. I mean for instance I want to download just the T1w images in MNI space (which is in the directory  ${StudyFolder}/${Subject}/MNINonLinear/T1w_restore.nii.gz of the structural preprocessed package) for a group of 1200 subject without downloading the whole structural preprocessed package. Is it possible to do so and how?

I would be grateful if you could help me...

Best,

Najme

[Glasser, Matt]

You can do that via REST calls for ConnectomeDB or using the AWS datasets.  You can look up how to do that on the HCP WIKI once it is back up. 

 

Matt.

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/ed610468-b7af-4716-b89d-3e9b5ce64136n%40humanconnectome.org.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[najme soleimani]

Dear Matt.,

Thanks for your quick reply. I used this link (https://wiki.humanconnectome.org/display/PublicData/How+To+Access+Subject+Data+via+REST) in order to download a specific data I need. The problem is I could download the data but it's only 4 kb !! I think something here went wrong because I could not even extract the data. I would appreciate if you could help me.

another question about AWS datasets, as I mentioned I want to download a specific data for a group of subjects so I filtered the data so the data should be downloaded. unfortunately, the filtering is not work when downloading. (I mean I selected for example a group of 500 subjects to be downloaded but when I want to download with the help of AWS it downloaded the specific file that I need for the whole 1200 subjects).

Thanks in advance...

BEST,

Najme

[Hodge, Michael]

Hi Najme,

I'm assuming the 4kb file that downloaded contains an error message indicating the problem.   We were having some filesystem issues this morning and over the weekend that likely affected your downloads.  Those have been recently resolved.  You might get a better result upon trying again.

Regarding AWS and filtering, I'm not sure what filtering you're using, but AWS isn't tied into the ConnectomeDB filtering, so filtering in ConnectomeDB wouldn't affect AWS downloads.  You could use something like s3fs (https://github.com/s3fs-fuse/s3fs-fuse) to mount the HCP_1200 release as a filesystem and access the subjects you want that way, like you're copying them from a filesystem.  I used this to mount the HCP_1200 folder in the hcp-openaccess bucket to my /mnt/s3/HCP_1200 folder:

s3fs hcp-openaccess:/HCP_1200 /mnt/s3/HCP_1200

Alternatively, you could use the AWS CLI (command line interface - https://aws.amazon.com/cli/) to copy files for specific subjects from the AWS bucket:

aws s3 cp s3://hcp-openaccess/HCP_1200/100307/MNINonLinear/T1w_restore.nii.gz .

I hope this helps.

Regards,

Mike

---

From: najme soleimani <najme.sol...@gmail.com>
Sent: Monday, December 20, 2021 9:41 AM
To: HCP-Users <hcp-...@humanconnectome.org>
Cc: Glasser, Matt <glas...@wustl.edu>
Subject: Re: [hcp-users] Downloading a particular file

 

* External Email - Caution *

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/2b389475-efe8-4652-ae64-47ea39f0a8f8n%40humanconnectome.org.

[najme soleimani]

Dear Mike,

Thanks for your great explanation. I really appreciate it.

Sorry for asking so many questions. I fixed the problem as you said and tried to download data by using curl tool and CREST as I mentioned earlier,  but I could download only 150 of subjects and a great number of subjects (like subject 101309 or 110411) was not downloaded. I searched these subjects in connectome db and there was no problem with the package. (I mean the data and package exists like all other subjects which were downloaded). I wonder where the problem is...

Thanks in advance.

BEST,

Najme

[Hodge, Michael]

Hi Najme,

We appear to be having further filesystem issues.  While hopefully our systems administrator will be able to fix soon, I'd recommend you use one of the AWS methods at this time for obtaining the files you're after.  That should be more reliable.  We're in the process of transitioning to a new filesystem and are experiencing stability issues as we make that transition.

Regards,

Mike

---

From: najme soleimani <najme.sol...@gmail.com>
Sent: Wednesday, December 22, 2021 8:07 AM
To: HCP-Users <hcp-...@humanconnectome.org>
Cc: Hodge, Michael <hod...@wustl.edu>

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/9916afd7-4f5d-4ea6-95a5-b65ab4873999n%40humanconnectome.org.