aiobotocore / aiohttp ClientConnectionError with many files requested

[jcpet...@gmail.com]

Hello, I'm trying to work out some way to use aioboto3 or aiobotocore successfully with python3.5.

I've written a few functions for pulling down a large number of files from s3, and I have successfully tested them and gotten them working on python3.6, with the following combination of libraries:

    python3.6.6

    aioboto3==4.1.2

    aiobotocore==0.9.4

    aiohttp==3.4.4

    boto3==1.9.7

    botocore==1.10.58

But to my dismay I found out that python3.5 is not supported by aiohttp 3.3.x, so I had to downgrade it, and have been attempting to use different combinations of libraries to see if I can get it to work. I am currently using the following combination:

    python3.6.6

    aioboto3==3.0.0

    aiobotocore==0.8.0

    aiohttp==3.1.3

    boto3==1.9.7

    botocore==1.10.12

Those combinations of libraries that do not crash due to missing arguments or similar, will fail on a large number of files (e.g., 4-5 files seems to work, but closer to 90-100 seems to fail).

They fail with a traceback similar to the following:

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aioboto3/s3/inject.py", line 30, in download_file

    await download_fileobj(self, Bucket, Key, open_file, ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aioboto3/s3/inject.py", line 73, in download_fileobj

    resp = await self.get_object(Bucket=Bucket, Key=Key)

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiobotocore/client.py", line 90, in _make_api_call

    operation_model, request_dict)

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 322, in _send_request

    request, operation_model, attempts)

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 419, in _get_response

    operation_model)

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 112, in convert_to_response_dict

    response_dict['body'] = await http_response.read()

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 815, in read

    self._body = await self.content.read()

  File "/home/jcp/.virtualenvs/colosseum/lib/python3.6/site-packages/aiohttp/streams.py", line 288, in read

    raise self._exception

aiohttp.client_exceptions.ClientConnectionError: Connection closed

I'm pretty confused as to how to proceed or debug these problems. Can anyone provide any guidance?