Move thousands of file from list using gsutil mv command

826 views
Skip to first unread message

R1

unread,
Aug 5, 2022, 5:22:38 AM8/5/22
to Google Cloud Dataproc Discussions
Hi guys , I want to move thousands of specific files parallely from GCS bucket using python or using command line inside python. I have GCS url of source aa well as destination in list. How do I run all of them in parallel like in 100s using python. I cannot use wild card because files are specific.

source_list=[ 'gs://bucket/folder1/obj1.txt', 'gs://bucket/folder1/obj2.txt' ....1000s]

destination= [ 'gs://bucket/folder2/obj1.txt', 'gs://bucket/folder2/obj2.txt' ....1000s]

Matias Coca

unread,
Aug 5, 2022, 8:20:16 AM8/5/22
to Google Cloud Dataproc Discussions
You need to use `gsutil` with `rsync`. Check these two links for more information


Regards.

Reply all
Reply to author
Forward
0 new messages