AIO (libaio) for async file access

[Michal Frackowiak]

Hi there,

would it be possible to combine libaio for asynchronous file access
(wrapping POSIX aio_*) with EventMachine? Would it make sense?

Some links:
http://www.twistedmatrix.com/pipermail/twisted-python/2007-December/016487.html
- similar thing for Python's Twisted, not complete
http://timetobleed.com/io-models-how-you-move-your-data-matters/ -
nice introduction to io concurrency
http://rubyforge.org/projects/rbaio/ - Win-only, abandoned, only aio-
ruby stuff I could find

[Michal Frackowiak]

Hi,

do you think it would be worth implementing aio_* wrappers for async
file access in EM? Did anyone try doing this? IMHO it would be a
really nice feature in EM.

Cheers,
Michal

[Aman Gupta]

Lourens has some AIO wrappers for ruby at http://github.com/methodmissing/aio

Aman

> --
>
> You received this message because you are subscribed to the Google Groups "EventMachine" group.
> To post to this group, send email to eventm...@googlegroups.com.
> To unsubscribe from this group, send email to eventmachine...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/eventmachine?hl=.
>
>
>

[Aman Gupta]

We've discussed this in the past, but imho, AIO is not really ready
for production use. On linux, AIO falls back to blocking primitives
when used with network i/o, and there are several limitations when
using it with disk i/o. What use case are you solve with AIO?

Aman

>
> Cheers,
> Michal

[Michal Frackowiak]

On Nov 23, 8:26 am, Aman Gupta <themastermi...@gmail.com> wrote:

>
> We've discussed this in the past, but imho, AIO is not really ready
> for production use. On linux, AIO falls back to blocking primitives
> when used with network i/o, and there are several limitations when
> using it with disk i/o. What use case are you solve with AIO?
>
>   Aman

One example use case:

I am using Thin (web server), using async processing. I would like to
prevent Thin from waiting for disk IO, while it could serve other
requests, thus increase concurrency.

The actual system is more complex than that, data on disk is heavily
accessed, but about 50% requests does not require any disk IO at all
(using cache).

But honestly, data integrity is much more important, so if aio_* have
some problems, they do not fit.

Thanks,

Michal

[Michal Frackowiak]

I can see you can attach a file descriptor via EM::attach, and use
notify_readable(writable) on it. I could not find too much information
about it though.

Could the current EM be used for async file reading/writing? Thanks in
advance.

Michal

[Aman Gupta]

No, calling select() or epoll() on a disk based fd always returns
readable/writable, and non-blocking operations are not possible. What
is your use-case, what kind of files are you dealing with? Are you
mostly reading or writing?

There are some tricks for reading.. you can EM.popen a process 'cat
filename'. For writing, you can buffer data and flush to disk every
few reactor ticks.

The other approach to disk i/o, that node.js takes, is using a
threadpool with the libeio library. EM does not currently support
libeio, but its something I have considered in the past.

Aman

>
> Michal
>
> --
>
> You received this message because you are subscribed to the Google Groups "EventMachine" group.
> To post to this group, send email to eventm...@googlegroups.com.
> To unsubscribe from this group, send email to eventmachine...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/eventmachine?hl=en.
>
>
>

[Tim Uckun]

> There are some tricks for reading.. you can EM.popen a process 'cat
> filename'. For writing, you can buffer data and flush to disk every
> few reactor ticks.
>

Would this be an appropriate mechanism for tailing multiple files?

[Aman Gupta]

I meant to say 'tail -f file', not 'cat file'. You can definitely call
EM.popen multiple times to tail files in parallel.

Aman

[Michal Frackowiak]

Thanks for the answers.

Starting shell processes is not really an options - I need to have
full access to the file fs, including seek, read, write.

I was experimenting a bit with notify_readable, and using
Kernel::select with small timeouts (and returning to EM loop if no
files fds are ready), but both approaches have terrible performance
compared to pure file.write on a non-loaded system, at a factor of 10
times less ios per second with short writes/reads.

I do not have a detailed analysis of expected IO, but so far it looks
that using blocking file IO is a good first approach. I will probably
go back to the subject once my project matures a bit and goes out of
stealth mode.

BTW: I cannot get http://github.com/methodmissing/aio working at all,
either MacOsX or Linux, so I did not really consider this.

Thanks again for all the comments.

Michal