aiohttp connection starvation leading to timeouts ?

[kristian....@gmail.com]

I *was* intending to split my list of 1,000,000 tasks into 10 lists of 100,000 items each, and send them off to 10 asyncio futures using this shared pool: 

conn = aiohttp.TCPConnector(limit=11)
async with aiohttp.ClientSession(connector=conn) as session:

Instead I managed to split it into 100,000 lists of 10 items because I'm an idiot.

After some time I would end up with these errors:

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 860, in start
    self._continue = None
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/helpers.py", line 586, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError


I'm assuming this is related to excessive waiting for connections to be available in the pool and that the pool might not be fair, or I'm being highly unfair by trying to allocate 100.000 connections from a pool of 10. It does not really matter. The problem is that I spent quite a lot of time chasing wild geese trying to track this down, because the error seems to indicate a problem with the remote system. I have not read the code, but it would appear to me that my code is timing out because I was not able to get a connection from the local connection pool within the specified timeout. Somehow I was interpreting this timeouterror to mean the remote system was timing out. Would it be possible to differentiate these two errors somewhat ?


Kristian