Google Cloud On-Premises Data Transfer - setting up transfer from PC

[Elizabeth Thompson]

I am using Docker (installed via Docker Desktop on Windows OS) and have used manual commands as per the instructions to create a pool and 3 agents to handle the transfer. When I create the transfer job in the console (or via CLI), I can't figure out if it's possible, and if so how to specify the POSIX path for the source. Ideally the source would be a network-attached storage (NAS) that's mounted to my current file system as a drive letter, but currently I can't get it to work even with files from my main computer, so I am likely missing some nuance. 

If I'm using this command to set up the agent(s), does it need to be run from within a Docker virtual env?  What does the "/:/transfer_root" actually contain and is there a way to see what the filesystem the agent has access to is?

docker run -d --ulimit memlock=64000000 --rm --volumes-from gcloud-config -v /:/transfer_root gcr.io/cloud-ingest/tsop-agent:latest --enable-mount-directory --project-id=PROJET-ID --hostname=$(hostname) --agent-pool=transfer_service_default

gcloud alpha transfer jobs create posix:///path/on/pc gs://bucket/ --source-agent-pool=POOL/ADDRESS

Thanks,

Liz

[Ajitesh Abhishek]

+ @Nicholas Hartunian + @Eric Li 

To see filesystem agent has access to - You can SSH in docker (with something like docker exec -ti container_id bash) and run `df -h`

--
© 2020 Google LLC. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Data Transfer Service Discussion Google Group (google-cloud-data...@googlegroups.com) to participate in discussions with other members of the Transfer Service community and the Transfer Service Team.
---
You received this message because you are subscribed to the Google Groups "google-cloud-data-transfer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-data-trans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-data-transfer-discuss/131be0d5-0bf7-4fbb-953f-55f6cf1c6a4fn%40googlegroups.com.

--

Regards,

Ajitesh
ajab...@google.com
Product Manager

[fiescocasasola]

Could you please share the documentation used to come up with the commands that you provided? Here is some information that you might find useful: 

• https://cloud.google.com/storage-transfer/docs/create-manage-transfer-gcloud#create-cloud-to-cloud 

• https://cloud.google.com/storage-transfer/docs/on-prem-agent-command-options

• https://stackoverflow.com/questions/41707573/how-does-docker-embedded-dns-resolver-work

• https://docs.docker.com/storage/

• https://cloud.google.com/storage-transfer/docs/managing-on-prem-agents

• https://cloud.google.com/storage-transfer/docs/on-prem-overview

[Elizabeth Thompson]

That did work for navigating the filesystem of the agent. 

I can see that C:\ is mounted on /transfer_root/run/desktop/mnt/host/c   --- assuming I had a test folder called "subset" in C:, the source folder should be identified as  /transfer_root/run/desktop/mnt/host/c/subset ?

Through the console and using the manual gcloud transfer command, I still get an error that the source cannot be found. I've read much of the documentation suggested in the other answer but am less familiar with docker, so perhaps I am still missing something.

Thanks,

Liz

[Eric Li]

Try it without the /transfer_root. Just set the source folder to: run/desktop/mnt/host/c/subset 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-data-transfer-discuss/c33e17d0-cf1a-46ea-8c92-f6e22892f67an%40googlegroups.com.

[Elizabeth Thompson]

That did the trick thank you for the advice!  Now it is stuck saying "IN_PROGRESS" but it could be because of the agent setup or that I started a gsutil sync which is currently running while I was trying to set this up.