rsync is slow in in copying huge folder with about 1996002 sub folders
20 views
Skip to first unread message
Phanindra Srungavarapu
unread,
Jul 25, 2016, 9:05:48 AM7/25/16
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to gce-discussion
Folder structure is like 000/001/001 to 002/450/999
Total estimate of folders is roughly about 1996002 each folder containing about 5 image files almost all images are less than 50K.
I'm using rsync to copy images from an external server to google could server in the process of migrating our data to google servers. We are moving all our servers to google. Network latency from our server location to our cloud computing engines is about 150ms. Rsync has been running for about 2 weeks, I'm running rsync with -azv --progress is there anyway I can improve the speed of the process??
George (Google Cloud Support)
unread,
Jul 25, 2016, 1:14:40 PM7/25/16
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to gce-discussion
Hello Phanindra,
If I correctly understood, you are copying your data to a GCE VM? if so, the speed depends on the Disk type and size and the instance type you are using.
You can try copying your data to Google Cloud Storage using the gsutil command with the -m flag as follows, to perform parallel synchronization: