Re: [gs-discussion] gsutil resumable downloads, detecting a dropped connection

[Google Cloud Storage Team]

Hi Timo,

You should be able to set the http socket timeout via the .boto config file, and it should affect the timeout on the connection. To do this, add a value for http_socket_timeout in the [Boto] section of your .boto config file.

If this doesn't work, will you tell us what you tried in a little more detail and what makes you think it didn't work? 

Best Regards,

Benson

Google Cloud Storage Team

On Wed, Mar 27, 2013 at 12:43 PM, tts <seppa...@gmail.com> wrote:

Hi

I would like to use the Google Cloud Storage for content delivery to mobile devices running Windows. The devices are connected to the network every now and then for short time periods (5-30 minutes). The content delivery should happen in the background without user intervention, tolerate the dropped connections and resume the transfer on the next opportunity. 

I have tested the gsutil command line tool, which supports resumable downloads. However, gsutil does not seem to react to a dropped connection so that it would return an error code. Instead it just keeps waiting forever. I traced this back to socket timeout handling of the boto library. A configuration option http_socket_timeout is available for boto, but that does not seem to do anything. 

I tried manually setting the socket timeout at gsutil/boto/boto/https_connection.py CertValidatingHTTPSConnection.connect, and that does seem to fire the socket timeout exception as expected when the network connection is dropped. 

I am not familiar with the development process of gsutil or boto. Should this be reported as a bug to either project, or is there perhaps some other mechanism already in place to react to the dropped connections? 

gsutil/gslib/commands/cp.py contains the following comments at the function _DownloadObjectToFile (line 1015):

      # If a custom test method is defined, call it here. For the copy command,

      # test methods are expected to take one argument: an open file pointer,

      # and are used to perturb the open file during download to exercise

      # download error detection.

Is there example code available on how to implement such a test method?

gsutil 3.25

python 2.7.2 with ssl

similar behaviour on OS X 10.8.2 and Windows 8 Pro

The lost network connection was simulated by "ifconfig en0 down" on Mac and by enabling the airplane mode on Windows.

Regards,

Timo

--
You received this message because you are subscribed to the Google Groups "Google Cloud Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gs-discussio...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Benson Kalahar]

Hi Timo,

After some careful inspection of the code, and of the boto commit logs on github, it looks like you're seeing an issue that was fixed in boto a month ago.  Since you're using gsutil version 3.25, that fix isn't in the copy of boto distributed with gsutil, but if you upgrade to gsutil 3.26, the timeout value you've set in your .boto config should propagate correctly and prevent the gsutil hang you're experiencing.  

Please let me know if this solves your issue. 

All the best,

Benson

Google Cloud Storage Team

On Wed, Mar 27, 2013 at 2:50 PM, tts <seppa...@gmail.com> wrote:

Benson,

Thanks for your reply.

I did indeed set the http_socket_timeout in the .boto config file. I tried to download a large file from a bucket in the Cloud Storage with the command:

gsutil -D cp gs://mybucket/test.bin .

The sanitized console output is attached.

When the transfer had been proceeding for a while, I terminated the network connection with "ifconfig en0 down". At that point gsutil froze even though I had defined a http_socket_timeout of 3 seconds. By pressing ctrl-c I got the stack trace that is visible on the last lines of the attached file. That shows that the process was waiting for additional data at Python SSL read function.

Then I altered the file gsutil/boto/boto/https_connection.py and added the following at line 110 in the CertValidatingHTTPSConnection.connect function:

if self.timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:

        sock.settimeout(self.timeout)

After this change, gsutil does not freeze after the network connection is terminated but quits when the timeout has elapsed. A debug message "SSLError: The read operation timed out" is shown.

In my opinion the reason is that the overridden CertValidatingHTTPSConnection.connect function does not pass the timeout value to the socket as it should. My two lines of code may not be compatible with Python 2.5 since the timeout was included from 2.6.

Regards,

Timo