Re: [gs-discussion] Large Uploads

Mike Schwartz (Google Storage Team)

unread,

Sep 15, 2012, 2:43:04 PM9/15/12

to gs-dis...@googlegroups.com

Hi,

We are investigating this problem and will follow up once we have information about it.

Thanks,

Mike

On Sat, Sep 15, 2012 at 10:12 AM, sgm <sogma...@gmail.com> wrote:

I'm trying to upload a big directory (1TB) with thousands of 'small' files (nohup gsutil -m cp -R -q /dir/* gs://bucket_name &). But after a couple of hours I can see the CPU usage skyrocketing to 100% and the network traffic dropping to 0 (for 24hrs). My guess is that gsutil is trying to compute an md5... but none of the files is bigger than 20mb. There is nothing useful on nohup.out. Any ideas or suggestions?

--

sgm

unread,

Sep 16, 2012, 4:50:35 AM9/16/12

to gs-dis...@googlegroups.com, gs-...@google.com

Thanks Mike! btw been doing some more tests and it seems to get stuck at "Catching up md5 for resumed upload" (on small files). feels like a leak.

sgm

unread,

Sep 20, 2012, 12:46:36 PM9/20/12

to gs-dis...@googlegroups.com, gs-...@google.com

It's working now with gsutil release 3.18. Thanks!

On Saturday, September 15, 2012 11:43:25 AM UTC-7, Mike Schwartz (Google Storage Team) wrote:

Reply all

Reply to author

Forward