Asyncio + file logging has sense?

[cr0hn cr0hn]

Hi folks,

After looking a lot on the internet I can't find an answer for that. I expose my doubt: 

Asyncio can't access to disk without blocking the event-loop. right? For example if I'm using aiohttp or Sanic, I can't write in a file in one of my end-points, without block the loop. Then... Can I use standard logging library with the FileHandler to log into a file without blocking the event loop?

Thanks in advance!

[INADA Naoki]

You're right. logging.FileHandler **may** block.

More precisely, FileHanlder calls flush, but not fsync or fdatasync.
So your application won't be blocked unless your application produce
massive logs. When it is blocked is up to your system (OS setting,
DISK I/O speed, etc...)

To avoid potential blocking, I recommend using PIPE.
For example, you can send your log to Apache rotatelogs through PIPE.

--
INADA Naoki <songof...@gmail.com>

[Guido van Rossum]

There's code in trio for treating the filesystem as async, and massive docs about when to use and when not. (Thanks Nathaniel!) We should consider adding that to asyncio, perhaps as a 3rd party package.

[Andrew Svetlov]

aiofiles already exist

--

Thanks,

Andrew Svetlov

[Guido van Rossum]

Is it stable enough to be merged into the stdlib? Is there anything we can learn from trio's version? I was impressed by the docs:
http://trio.readthedocs.io/en/latest/reference-io.html#asynchronous-filesystem-i-o
e.g. "Why is async file I/O useful? The answer may surprise you"

--

--Guido van Rossum (python.org/~guido)

[Yury Selivanov]

I'm not sure if aiofiles is stable or not, but I would like the actual implementation to be slightly different.  aiofiles simply proxies all file IO operations through a thread pool, but there are better alternatives in some cases.  Using a thread pool is totally fine for vanilla asyncio, but there should be a way for projects like uvloop/twisted to inject their own implementation of asynchronous file IO.

To do that we probably need to define an abstract file IO interface and add a method to event loop to return an object that implements that interface.  For instance:


    # in asyncio:

    class asyncio.AbstractFileIOImplementation:

        async def open(self):
             raise NotImplementedError

    async def open(...):
        return await get_current_loop(). get_file_io_implementation().open(...)


    # in uvloop or twisted or ...:

    class uvloop.FileIO(asyncio.AbstractFileIOImplementation):
        # ... implementation of "open()"

    class uvloop.Loop:
        def get_file_io_implementation(self):
            return self


    # later in user code:

    async def main():
        file = await asyncio.open(...)

A link to this code snippet in case email screws up formatting: https://gist.github.com/1st1/a1936d2f283c1e2f1848c4ed42634ebf

Thanks,
Yury

[Andrew Svetlov]

Sounds like a good target for Python 3.8

IMHO aiofiles is stable in terms of "not many api changes in last years".
But I unlike the implementation: it uses heavy metaprogramming. Saves lines of code but makes understanding and debugging much harder.

--

Thanks,

Andrew Svetlov

[Ludovic Gasc]

Hi,

aiofiles works pretty well, it's rock solid, we use it since the beginning of aiofiles.

Nevertheless, for me, it makes sense to have a builtin solution for AsyncIO, handling a file is a common operation that is already builtin with the sync Python standard library.

About the original question for logging, the easiest way to avoid to block the event loop is to delegate to a local rsyslog or journald:

Even if the storage has an issue, it won't block your daemon for logging.

Have a nice week.

--

Ludovic Gasc (GMLudo)