Fatal error on SSL transport

[Al Johri]

Hi All,

I'm getting a strange SSL error nondeterministically and would love some help debugging it.

2016-02-04 15:21:33.516 ERROR base_events - default_exception_handler: Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x1221377f0>
transport: <_SelectorSocketTransport fd=37 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/Users/johria/.pyenv/versions/3.5.1/lib/python3.5/asyncio/sslproto.py", line 627, in _process_write_backlog
    ssldata = self._sslpipe.shutdown(self._finalize)
  File "/Users/johria/.pyenv/versions/3.5.1/lib/python3.5/asyncio/sslproto.py", line 154, in shutdown
    ssldata, appdata = self.feed_ssldata(b'')
  File "/Users/johria/.pyenv/versions/3.5.1/lib/python3.5/asyncio/sslproto.py", line 218, in feed_ssldata
    self._sslobj.unwrap()
  File "/Users/johria/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 637, in unwrap
    return self._sslobj.shutdown()
ssl.SSLError: [SSL] called a function you should not call (_ssl.c:2095)

This is about a ~100 line gist from part of my application where the problem occurs: https://gist.github.com/AlJohri/6d4f21d7e57bd672b25a

I'll try to come up with a minimal example if needed but because it occurs intermittently that may be a little difficult.

Essentially, it uses aiohttp.get to download the HTML content of about ~70000 news articles. It passes the articles in chunks of 1000 to the asyncio loop. It then basically pickles a transformed version of the response to disk.

I would love any feedback on ways in which I can make this code more idiomatic async code. For example, the error handling is getting out of hand - is there an easier way to catch the errors? I'm mainly looking at aiohttp.errors.ServerDisconnectedError, aiohttp.errors.ClientResponseError, asyncio.TimeoutError, asyncio.CancelledError.

Thanks!