How do I increase the timeout for imaplib requests?

[John Del Rosario]

I'm using imaplib to query Gmail's IMAP, but some requests are taking more than 60 seconds to return. This is already done in a task, so I should have a full 10 minutes to do the request, but my tasks are failing due to the 60 second limit on urlfetch.

I've tried setting urlfetch.set_default_fetch_deadline(600), but it doesn't seem to do anything.

Here's a stacktrace:

The API call remote_socket.Receive() took too long to respond and was cancelled.
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 760, in uid
    typ, dat = self._simple_command(name, command, *args)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 897, in _command_complete
    typ, data = self._get_tagged_response(tag)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 999, in _get_tagged_response
    self._get_response()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 916, in _get_response
    resp = self._get_line()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1009, in _get_line
    line = self.readline()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1171, in readline
    return self.file.readline()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/socket.py", line 445, in readline
    data = self._sock.recv(self._rbufsize)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ssl.py", line 301, in recv
    return self.read(buflen)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ssl.py", line 220, in read
    return self._sslobj.read(len)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 864, in recv
    return self.recvfrom(buffersize, flags)[0]
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 903, in recvfrom
    apiproxy_stub_map.MakeSyncCall('remote_socket', 'Receive', request, reply)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall
    rpc.CheckSuccess()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call remote_socket.Receive() took too long to respond and was cancelled.

[Vinny P]

On Wed, Oct 23, 2013 at 8:39 PM, John Del Rosario <jo...@collabspot.com> wrote:

but my tasks are failing due to the 60 second limit on urlfetch.

I've tried setting urlfetch.set_default_fetch_deadline(600), but it doesn't seem to do anything.

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 864, in recv

This is not a URLFetch problem. IMAPlib doesn't connect via HTTP (which URLFetch handles), it connects via IMAPS port 993. This means that the failure is in the sockets API. Notice that the stack trace makes reference to sockets, not urlfetch. You need to tell the socket to stay open longer. I don't believe imaplib supplies an easy way to increase the socket timeout, so you may need to dig into imaplib source and increase the timeout there.  

What IMAP operations are you calling that takes 60+ seconds for the server to respond? I have a Java application that connects to Gmail via IMAP and other external mail servers via POP3, and I have always received fast responses.

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[John Del Rosario]

I'm doing a SEARCH  with "HEADER In-Reply-To <rfc822_msg_id>" to get a message's replies. I'm not sure how to verify it, but I'm guessing maybe it's happening for emails that have a lot of replies already? It's only happening on a couple of emails.

Would you know if it's possible to only get 1 message from the SEARCH? I only need one. And I don't need the messages' bodies either. So if it's possible to limit the response, maybe that will help with the issue.

Also, would doing multiple requests to the same IMAP server simultaneously have any undesirable results?

Thanks,
John

[Vinny P]

On Thu, Oct 24, 2013 at 10:03 AM, John Del Rosario <jo...@collabspot.com> wrote:

I'm doing a SEARCH  with "HEADER In-Reply-To <rfc822_msg_id>" to get a message's replies. I'm not sure how to verify it, but I'm guessing maybe it's happening for emails that have a lot of replies already? It's only happening on a couple of emails. 

Would you know if it's possible to only get 1 message from the SEARCH? I only need one. And I don't need the messages' bodies either. So if it's possible to limit the response, maybe that will help with the issue.

I don't believe IMAP supports limiting search results by the number of results. You would have to limit by another criteria, such as SENTSINCE (emails sent after a certain date), subject, etc.

What you can do instead is use Gmail's IMAP extensions. Gmail IMAP offers the X-GM-THRID command ( https://developers.google.com/gmail/imap_extensions#access_to_the_gmail_thread_id_x-gm-thrid ) which you can use to retrieve whole threads (original email, plus replies). And imaplib allows the usage of IMAP extensions via xatom ( http://docs.python.org/2/library/imaplib.html#imaplib.IMAP4.xatom ). Unfortunately you may have to do some parsing of the results, but it shouldn't be too bad.

On Thu, Oct 24, 2013 at 10:03 AM, John Del Rosario <jo...@collabspot.com> wrote:

Also, would doing multiple requests to the same IMAP server simultaneously have any undesirable results?

Unlikely, since current mail clients also frequently open up multiple connections to mail servers.

Note that Gmail allows a maximum of 15 simultaneous connections, so don't open too many (note https://support.google.com/mail/answer/97150?hl=en ).

 

[John Del Rosario]

After a bit of digging in the AppEngine source and Python's imaplib and socket library, it seems by default sockets have no timeouts. So it's weird that my requests are timing out at exactly 60 seconds.

Regarding the X-GM-THRID command, how would I generate the GM-THRID from a message UID or RFC822 ID? (I don't have the GM-THRID at hand, and it's not possible for me to get it from the browser url and DOM)

Thanks,
John

[Vinny P]

On Thu, Oct 24, 2013 at 8:44 PM, John Del Rosario <jo...@collabspot.com> wrote:

Regarding the X-GM-THRID command, how would I generate the GM-THRID from a message UID or RFC822 ID? (I don't have the GM-THRID at hand, and it's not possible for me to get it from the browser url and DOM)

You don't generate it. The thread ID is supplied when you use the FETCH command with X-GM-THRID as an argument. See here for an example of how it works: https://developers.google.com/gmail/imap_extensions#access_to_the_gmail_thread_id_x-gm-thrid and here: http://yuji.wordpress.com/2011/06/22/python-imaplib-imap-example-with-gmail/ (scroll down to where it says Fetches).

[John Del Rosario]

Ah yes. But the issue is that I don't know the GM-THRID by the time I trigger my task. All I have to go by are the email's subject/recipients/date, and with that I do a SEARCH query to get the UID and RFC822 ID.

[Vinny P]

On Sun, Oct 27, 2013 at 8:57 PM, John Del Rosario <jo...@collabspot.com> wrote:

Ah yes. But the issue is that I don't know the GM-THRID by the time I trigger my task. All I have to go by are the email's subject/recipients/date, and with that I do a SEARCH query to get the UID and RFC822 ID. 

But you can extract X-GM-THRID using the ID of the email. Call SEARCH and get the message ID of the original email using the data you've stored. Then call FETCH [ID] (X-GM-THRID) to get the thread id. Then call SEARCH again with the X-GM-THRID argument to extract all replies.

Of course, this is a bit roundabout - the easier way would be to record the thread id when you're originally fetching the emails to store, prior to calling the task.

[John Del Rosario]

Oh, you're right. I missed that in the docs. Gonna try that out. Thanks!