Using io-queue pressure at the application layer

[Travis Downs]

Is there a recommended way to use io queue pressure at the application layer?

Currently seastar does a lot of hard to work to avoid submitting IOs to the device (well, to the kernel) if it would not help with throughput, but the primary benefits seem to be (a) applying IO priorities more effectively and (b) allowing IO to be canceled for a longer period of time.

However, one would naturally also like to take advantage of the fact that some queues are still queued-not-submitted at the application layer, e.g., to extend or replace existing IOs as new requests come in that would modify them.

Is there a standard approach for that? Currently it seems like to do that we could build *another* IO queue at the application layer, to avoid submitting them to seastar until we are sure they will be promptly sent to the device, but this seems redundant and sub-optimal.

[Avi Kivity]

On Mon, 2023-08-14 at 13:58 -0700, Travis Downs wrote:

Is there a recommended way to use io queue pressure at the application layer?

Currently seastar does a lot of hard to work to avoid submitting IOs to the device (well, to the kernel) if it would not help with throughput, but the primary benefits seem to be (a) applying IO priorities more effectively and (b) allowing IO to be canceled for a longer period of time.

However, one would naturally also like to take advantage of the fact that some queues are still queued-not-submitted at the application layer, e.g., to extend or replace existing IOs as new requests come in that would modify them.

Is there a standard approach for that?

There isn't a way, but I had the same thoughts - when a queue is exhausted, call a callback to allow the application to generate work just in time. It's not easy since you don't want to generate work outside CPU scheduling, but you only know you're out of work during poll time. Perhaps instead of "out of work" you'd check for "you're soon going to be out of work" for some value of soon (say two task quotas).

I also want to do the same thing for CPU: when a scheduling group is empty, call a callback to generate more work. This requires nested scheduling groups however to be really effective.

 Currently it seems like to do that we could build *another* IO queue at the application layer, to avoid submitting them to seastar until we are sure they will be promptly sent to the device, but this seems redundant and sub-optimal.

You currently don't have a way to tell when the I/O will be promptly be sent to the device. Even with my idea, you'd still need your own queue (but it would be a queue of high-level work items, not I/O).

One advantage is that you can keep memory for queued requests down by generating/allocating buffers at the last minute.