File Compression

[Cathy]

I uploaded a 7zip file to my bucket, but am unable to unzip.  Now, I'm trying to upload it using the gsutil -z option.  I'm getting an error message "Inadequate tempspace available to compress".  

[Mike Schwartz (Google Storage Team)]

Hi Cathy,

gsutil compresses files to a temporary directory before uploading them when you use the gsutil cp -z option. The default directory it picks may not be on a partition that has the largest space available, but you can change the directory it uses - please see the help text in the "CHANGING TEMP DIRECTORIES" section of "gsutil help cp".

Mike

On Tue, Nov 6, 2012 at 6:55 AM, Cathy <itdevtea...@gmail.com> wrote:

I uploaded a 7zip file to my bucket, but am unable to unzip.  Now, I'm trying to upload it using the gsutil -z option.  I'm getting an error message "Inadequate tempspace available to compress".  

--
 
 

[Cathy]

Thank you for the response. What is the ratio?  I have a 300GB file I want to compress and upload.  Approximately how much temp space will I need?

[Mike Schwartz (Google Storage Team)]

Hi Cathy,

It's hard to predict how much your data will compress; compression ratios are very data-dependent. Text often compresses by 90%, while binary data like images typically doesn't compress at all, or may even have negative compression. I recommend that you try using gsutil cp -z on a small sample of your data and looking at the size of the data after it's been uploaded (using gsutil ls -l).

As for how much space you'll need, it depends on how big the files are, and whether you use gsutil -m do do parallel uploads. gsutil will compress each file before uploading and then remove the temp copy the compressed data after it has been uploaded.

Mike

--