readuntil for big file raise IncompleteReadError

[Klutz ExNW]

Hey,

I am trying to read big file from remote server (about 4MB) and i receive: asyncio.streams.IncompleteReadError: 2099450 bytes read on a total of None expected bytes

.

Now according to what i read when this happens using readuntil you get the output till then in the exception.partial, and after that you should keep using readuntil/readlines till the end.

but after i retry using readuntil it raise Exception agian: asyncio.streams.IncompleteReadError: 0 bytes read on a total of None expected bytes

I made a text file called test.txt what is just huge file containt nothing but the letter "a" and tried to read it

Can you please take a look at this?

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Traceback (most recent call last):
  File "__init__.py", line 14, in run_client
    response = await stdout.readuntil([">", "#"])
  File "/usr/local/lib/python3.7/dist-packages/asyncssh/stream.py", line 153, in readuntil
    return await self._session.readuntil(separator, self._datatype)
  File "/usr/local/lib/python3.7/dist-packages/asyncssh/stream.py", line 558, in readuntil
    raise asyncio.IncompleteReadError(buf, None)
asyncio.streams.IncompleteReadError: 2099450 bytes read on a total of None expected bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "__init__.py", line 34, in <module>
    asyncio.get_event_loop().run_until_complete(run_client(config))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "__init__.py", line 18, in run_client
    response = await stdout.readuntil([">", "#"])
  File "/usr/local/lib/python3.7/dist-packages/asyncssh/stream.py", line 153, in readuntil
    return await self._session.readuntil(separator, self._datatype)
  File "/usr/local/lib/python3.7/dist-packages/asyncssh/stream.py", line 558, in readuntil
    raise asyncio.IncompleteReadError(buf, None)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of None expected bytes

import asyncssh

import asyncio

async def run_client(config):

    async with asyncssh.connect(**config) as conn:

        stdin, stdout, stderr = await conn.open_session(

            term_type="Dumb"

        )

        response = await stdout.readuntil([">", "#"])

        print(response)

        stdin.write("cat test.txt\n")

        try:

            response = await stdout.readuntil([">", "#"])

            print(response)

        except BaseException as err:

            print(err.partial)

            response = await stdout.readuntil([">", "#"])

            print(response)

config = {

            "host": "localhost",

            "port": 22,

            "username": "root",

            "password": "root",

            "known_hosts": None,

            "client_keys": None,

            "keepalive_interval": 0,

        }

try:

    asyncio.get_event_loop().run_until_complete(run_client(config))

except (OSError, asyncssh.Error) as exc:

    sys.exit('SSH connection failed: ' + str(exc))

[Ron Frederick]

Thanks for the report! I’ve been able to reproduce the problem here. The issue is that the readuntil() function doesn’t resume reading after returning the partial data. So, if you attempt to call readuntil() again, it still thinks the stream is paused and doesn’t wait to read any additional bytes. This can be corrected with the following change in stream.py:

diff --git a/asyncssh/stream.py b/asyncssh/stream.py

index f722643..28a5d5f 100644

--- a/asyncssh/stream.py

+++ b/asyncssh/stream.py

@@ -555,6 +555,7 @@ class SSHStreamSession:

                 if self._read_paused or self._eof_received:

                     recv_buf[:curbuf] = []

                     self._recv_buf_len -= buflen

+                    self._maybe_resume_reading()

                     raise asyncio.IncompleteReadError(buf, None)

 

                 await self._block_read(datatype)

This is enough to return more data to you, but there’s also an issue where your code wouldn’t handle that second call to readuntil() also returning an exception after reading more data. With 4 MB of data or more, you'd still run into a problem. This can easily be corrected by doing this in a loop, though. For instance, you could do something like:

async def read_response(reader, separator):

    response = []

    

    while True:

        try:

            response.append(await reader.readuntil(separator))

            break

        except asyncio.IncompleteReadError as exc:

            if exc.partial:

                response.append(exc.partial)

            else:

                break

    return "".join(response)

async def run_client():

    async with asyncssh.connect("localhost") as conn:

        stdin, stdout, stderr = await conn.open_session(term_type="dumb")

        print(await read_response(stdout, [">", "#"]))

        stdin.write("cat /tmp/test.txt\n")

        print(await read_response(stdout, [">", "#"]))

        stdin.write("exit\n")

        print(await read_response(stdout, [">", "#"]))

In this example, the entire contents of test.txt will be returned in the second call to read_response(), even though multiple calls to readuntil() were needed.

Note that this works for one-character separator strings, but you could run into a problem using a multi-character separator if the stream was paused right in the middle of reading one of the separators. In that case, the presence of the separator might be missed and it would be up to you to reassemble the pieces of the stream yourself to see if that happened.

-- 
Ron Frederick
ro...@timeheart.net

[Ron Frederick]

The fix below is now checked into the AsyncSSH “develop” branch and will be included in the next release.

Thanks again!

[Klutz ExNW]

Thank you for the feedback, This script was just so you can reproduce the problem.

בתאריך יום רביעי, 19 בפברואר 2020 בשעה 06:40:33 UTC+2, מאת Ron Frederick: