Read big file

[Ilya Chistyakov]

Hi All!

I have some problem. I want to read a large file and process its contents line by line. But my code loads the entire file, and it does not fit in the RAM. Below is a sample code. What can I do about it?

async with asyncssh.connect('localhost', known_hosts=None) as conn:
    result = await conn.run(
        'cat /var/log/very-big-log.log'
    )
    ids = set()
    for line in result.stdout.split('\n'):
        kv = dict(
            pair.split('=')
            for pair in line.split('\t')
        )
        ids.add(kv['id'])

Many thanks for any help!

[Ron Frederick]

Hi Ilya,

Instead of using the run() function here, you’re going to want to use create_process(), so you can read from stdout incrementally. In fact, since you are reading a line at a time, you can directly use the SSHReader corresponding to stdout as an async iterator which will return one line of output at a time. Here’s an example:

    async with asyncssh.connect('localhost', known_hosts=None) as conn:

        async with conn.create_process(‘cat /var/log/very-big-log.log') as proc:

            async for line in proc.stdout:

                ...do whatever you need to do for each line...

One note: the variable “line” here will get the entire line, including the ‘\n’ at the end if there is one. So, you may want to operate on line.strip(‘\n’) if you don’t want that as part of your result.

-- 
Ron Frederick
ro...@timeheart.net

[Ilya Chistyakov]

Thank you very much!

пт, 14 июн. 2019 г. в 07:47, Ron Frederick <ro...@timeheart.net>:

--

С уважением, Илья.