Request for Command-Line Download Instructions for AABC Release 1 and HCP-YA Datasets

[박정훈]

Dear HCP/AABC Support Team,

My name is Junghoon Justin Park, a PhD student at Seoul National University, Korea.
I am writing to request guidance on downloading large-scale datasets from ConnectomeDB powered by BALSA to a remote server.

• Background:

Due to the substantial size of the datasets I need (20+ TB in total), downloading to a local PC is not feasible. I have IBM Aspera Connect installed on both my browser and my lab's online server (IBM Aspera Connect Linux 3.9.9), and I would like to download the data directly to the server via command line.

• Requested Datasets:

1) Aging Adult Brain Connectome (AABC) Release 1 (all subjects):
- ASL Preprocessed Recommended
- CIFTI Multi-modal IDPs Recommended
- Concatenated fMRI Preprocessed Recommended
- Diffusion Preprocessed Recommended
- Resting State fMRI Preprocessed Recommended
- Structural Preprocessed Recommended
- Task fMRI Preprocessed Recommended

2) HCP-Young Adult 2025 (all subjects):
- Diffusion 3T Preprocessed Recommended
-  Diffusion 7T Preprocessed Recommended
-  Movie fMRI 7T Preprocessed Recommended
-  Resting State fMRI 3T Preprocessed Recommended
-  Resting State fMRI 7T Preprocessed Recommended
-  Retinotopy fMRI 7T Preprocessed Recommended
-  Structural Preprocessed Recommended for 3T and 7T
-  Task fMRI 3T Recommended

• Request:

Could you please provide step-by-step instructions for downloading these datasets via command line (ascp), including any necessary transfer parameters, authentication tokens, or scripts?

I have registered for a BALSA account and agreed to the Data Use Terms. Our research team is fully committed to adhering to all responsibilities for secure, ethical usage and storage of these data. The datasets will be used solely for scientific research purposes.


Thank you very much for your time and assistance.
Please let me know if you need any additional information.


Sincerely, 

Justin

[Elam, Jennifer]

Hi Justin,

Command line options for downloading from BALSA are not functional with our older Aspera server. We are planning to Implement one or more alternatives to the current GUI-based Aspera Connect in the next couple of months, but we don’t have it available yet. What we recommend for now is to mount the target server filesystem from a computer on the same network/in the same building that does have a GUI/browser to interact with Aspera.

Best,

Jenn

---

From: 박정훈 <utopi...@snu.ac.kr>
Sent: Thursday, January 1, 2026 7:38 PM
To: hcp-...@humanconnectome.org <hcp-...@humanconnectome.org>
Cc: ­차지욱 / 부교수 / 심리학과 <conne...@snu.ac.kr>
Subject: [hcp-users] Request for Command-Line Download Instructions for AABC Release 1 and HCP-YA Datasets

 

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CAJQeMMe9o5F75GuOC3CYsxWKhVvSYoYuybfUE61ijdeBow3n3Q%40mail.gmail.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Frank Yeh]

Hi Justin,

    I figured out a way to download it using command line so that the data could be transferred to the cluster directly without downloading to desktop and reupload it.
   The process is a bit complicated, and you may paste the following to ChatGPT or Gemini to guide you the process.

Downloading AABC from BALSA via command line (HPC / cluster)

---

1) Confirm Aspera CLI is available on your cluster

You need the IBM Aspera command-line client (ascp). On some clusters it is provided as a module:

module load aspera-connect/3.11.0.5
which ascp
ascp -V

If which ascp fails, ask your admin to install Aspera or provide a module.

---

2) Obtain the “payload” (transfer spec) from BALSA

BALSA uses Aspera Connect and provides a JSON “transfer spec” (payload) that contains the file paths, host/user, ports, token, cookie, and SSH fingerprint.

1. Go to the AABC project page on BALSA and select the files you want.

2. Open Chrome DevTools → Network tab.

3. Click Download (this triggers Aspera Connect and may ask you to choose a local destination folder). 

4. In DevTools → Network, find the request that starts the transfer (often called start).

5. Open that request and copy the Request Payload / payload JSON.

6. Save it on the cluster as a file, e.g. payload.txt.

Note: tokens can expire. If your cluster download fails with auth errors, regenerate a fresh payload.

---

3) Copy the Aspera Connect token-auth key to the cluster

Aspera Connect uses an “SSH-bypass” key named:

• aspera_tokenauth_id_rsa (passphrase-protected)

On Windows it is typically located at:

C:\Users\<USERNAME>\AppData\Local\Programs\IBM\Aspera Connect\etc\aspera_tokenauth_id_rsa

Copy that file to the cluster:

mkdir -p ~/.ssh chmod 700 ~/.ssh # copy file to ~/.ssh/aspera_tokenauth_id_rsa chmod 600 ~/.ssh/aspera_tokenauth_id_rsa

---

4) Retrieve the passphrase used by Aspera Connect (Windows)

The key is encrypted; Aspera Connect provides the passphrase to ascp at runtime via the environment variable ASPERA_SCP_PASS.

1. Start an AABC download on Windows so ascp.exe is actively running.

2. Open Sysinternals Process Explorer (Google this program and download it. This is free)

3. Locate the running ascp.exe process.

4. Open its Properties → Environment tab.

5. Find ASPERA_SCP_PASS and copy its value.

On the cluster, store it in a protected file:

printf '%s\n' 'PASTE_PASSPHRASE_HERE' > ~/.ssh/aspera_pass chmod 600 ~/.ssh/aspera_pass

Security note: Treat this like a password. Keep it private (chmod 600) and don’t paste it into shared documents.

---

5) Run the download on the cluster (recommended via SLURM)

For large downloads, avoid running on the login node.
Use sbatch (or an interactive compute job if your site prefers that).

A) Save this as download_tags.sh

#!/usr/bin/env bash
# Aspera Download Script (token + cookie + passphrase + tags64)
# Usage:
#   ./download_tags.sh payload.txt /path/to/download_dir
#
# Assumptions:
#   - tokenauth key:  $HOME/.ssh/aspera_tokenauth_id_rsa
#   - passphrase:     $HOME/.ssh/aspera_pass
#   - payload.txt is a fresh BALSA transfer payload

set -euo pipefail

module load aspera-connect/3.11.0.5

PAYLOAD="${1:?Usage: $0 payload.txt [/path/to/download_dir]}"
DEST="${2:-$PWD}"
mkdir -p "$DEST" "$DEST/log"

ASPERA_KEY="$HOME/.ssh/aspera_tokenauth_id_rsa"
[[ -f "$ASPERA_KEY" ]] || { echo "ERROR: missing key: $ASPERA_KEY" >&2; exit 1; }
chmod 600 "$ASPERA_KEY" 2>/dev/null || true

[[ -f "$HOME/.ssh/aspera_pass" ]] || { echo "ERROR: missing passphrase file: ~/.ssh/aspera_pass" >&2; exit 1; }
export ASPERA_SCP_PASS="$(< ~/.ssh/aspera_pass)"

python3 - "$PAYLOAD" <<'PY'
import json, base64, uuid, sys, pathlib

p = pathlib.Path(sys.argv[1])
d = json.loads(p.read_text())
ts = d["transfer_specs"][0]["transfer_spec"] if "transfer_specs" in d else d

host   = ts["remote_host"]
user   = ts["remote_user"]
token  = ts.get("token","")
cookie = ts.get("cookie","")
cipher = ts.get("cipher","aes-128").replace("-","")  # aes-128 -> aes128
ssh_p  = int(ts.get("ssh_port",33001))
fasp_p = int(ts.get("fasp_port",ssh_p))
sshfp  = ts.get("sshfp","")

tags64 = base64.b64encode(
  json.dumps({"aspera":{"xfer_id":str(uuid.uuid4()),"xfer_retry":0}}).encode()
).decode()

open("file_list.txt","w").write("".join(
  item["source"] + "\n" for item in ts.get("paths",[]) if "source" in item
))

with open("vars.sh","w") as f:
  f.write(f'export ASPERA_SCP_TOKEN={token!r}\n')
  if cookie: f.write(f'export ASPERA_SCP_COOKIE={cookie!r}\n')
  f.write(f'ASPERA_HOST={host!r}\nASPERA_USER={user!r}\nASPERA_CIPHER={cipher!r}\n')
  f.write(f'ASPERA_SSH_PORT={ssh_p}\nASPERA_FASP_PORT={fasp_p}\n')
  f.write(f'ASPERA_SSHFP={sshfp!r}\nASPERA_TAGS64={tags64!r}\n')
PY

source vars.sh
[[ -s file_list.txt ]] || { echo "ERROR: file_list.txt is empty (no paths in payload)" >&2; exit 2; }

echo "Token set? $([[ -n "${ASPERA_SCP_TOKEN:-}" ]] && echo yes || echo no)"
echo "Cookie set? $([[ -n "${ASPERA_SCP_COOKIE:-}" ]] && echo yes || echo no)"
echo "Pass set?  $([[ -n "${ASPERA_SCP_PASS:-}"  ]] && echo yes || echo no)"
echo "Host/User: $ASPERA_USER@$ASPERA_HOST"
echo "Dest:      $DEST"
echo

ascp -q --mode=recv \
  --user="$ASPERA_USER" --host="$ASPERA_HOST" \
  -P "$ASPERA_SSH_PORT" -O "$ASPERA_FASP_PORT" \
  -c "$ASPERA_CIPHER" -k 2 --policy=fair \
  --precalculate-job-size \
  --file-manifest=text --file-manifest-path="$DEST/file-manifest" \
  --tags64="$ASPERA_TAGS64" \
  ${ASPERA_SSHFP:+--check-sshfp="$ASPERA_SSHFP"} \
  -i "$ASPERA_KEY" \
  --file-list=file_list.txt \
  -L "$DEST/log" \
  "$DEST"

rm -f vars.sh
echo "Done."

Make it private/executable:

chmod 700 download_tags.sh

B) Submit as a SLURM job (recommended)

Save as aspera_download.sbatch:

#!/usr/bin/env bash
#SBATCH -J aspera_dl
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 12:00:00
#SBATCH -p RM
#SBATCH -o aspera_dl.%j.out
#SBATCH -e aspera_dl.%j.err

set -euo pipefail
cd "$SLURM_SUBMIT_DIR"

DEST="/scratch/$USER/AABC"
mkdir -p "$DEST"

./download_tags.sh payload.txt "$DEST"

Submit:

sbatch aspera_download.sbatch

---

Troubleshooting tips

• If it fails with authentication errors, your token likely expired → regenerate a fresh payload.

• If compute nodes cannot reach the Aspera host/port (firewall), you may need a transfer node or a throttled login-node download.

• Logs are in: DEST/log/

---

[Harms, Michael]

 

Awesome!  Thanks for sharing.

 

-- 

Michael Harms, Ph.D.

-----------------------------------------------------------

Professor of Psychiatry

Washington University School of Medicine

Department of Psychiatry, Box 8134

660 South Euclid Ave.

St. Louis, MO  63110

To view this discussion visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/df6a6628-136c-410d-9d0c-b3b56cf2533en%40humanconnectome.org.