up to 10X I/O throughput boost with the BFQ I/O scheduler

[Paolo Valente]

Hi,
I'm Paolo, main developer of the BFQ I/O scheduler [1], and this is to
share some information on BFQ.

Almost ridiculous to say here, isolation is fundamental with
containers. This also applies to I/O bandwidth. In this respect, as
many of you probably know, existing techniques for isolating I/O
bandwidths cause severe underutilization with common workloads [2]
(utilization is often around only 10%).

So in the last years I've worked a lot on BFQ to address this issue,
and now BFQ recovers this utilization loss almost completely. As a
result, BFQ gives from 5x to 10X throughput boost with containers (as
well as with multiple groups, virtual machines and so on) [2].

BFQ has been upstreamed since Linux 4.12, so why don't you consider it
as a simple solution to greatly boost performance? I'd be happy to
help in every respect.

Thanks,
Paolo

[1] https://algo.ing.unimo.it/people/paolo/BFQ/
[2] https://www.linaro.org/blog/io-bandwidth-management-for-production-quality-services/

[Benjamin Gilbert]

On Thu, Jul 18, 2019 at 4:07 AM Paolo Valente <paolo....@linaro.org> wrote:

BFQ has been upstreamed since Linux 4.12, so why don't you consider it
as a simple solution to greatly boost performance?  I'd be happy to
help in every respect.

Hi Paolo,

Container Linux already ships the BFQ scheduler as an option.  Are there barriers to using it?

--Benjamin Gilbert

[Paolo Valente]

> Il giorno 31 lug 2019, alle ore 18:37, Benjamin Gilbert <bgil...@redhat.com> ha scritto:
>
> On Thu, Jul 18, 2019 at 4:07 AM Paolo Valente <paolo....@linaro.org> wrote:
> BFQ has been upstreamed since Linux 4.12, so why don't you consider it
> as a simple solution to greatly boost performance? I'd be happy to
> help in every respect.
>
> Hi Paolo,
>

Hi Benjamin,

> Container Linux already ships the BFQ scheduler as an option. Are there barriers to using it?
>

Unfortunately the problem is in the default.

Last year Jens Axboe decided to hardwire mq-deadline as default I/O
scheduler, with no chance any longer to change the default at boot
(I've appended an explanation for this choice at the end of this
message).

So, even if BFQ is available in the kernel, the burden to switch to
BFQ, from userspace, is left to either the user-space part of the
operating system, or the final users. If your OS sticks to the current
default, then, as you can imagine, 99.99% of your users will naturally
interpret it as a sane default, and will stick to it too.

So, to actually use BFQ, you need the extra step of changing the
current default, or at least to inform your users about the I/O
throughput and low latency they are currently losing, because of lack
of proper I/o control. I'm ok also with this second, milder option,
if you find it less disruptive as a first move. The core issue is
just that the current state of affairs means leaving BFQ and its
benefits in the basement.

Thanks,
Paolo


Appendix

Why BFQ is not the default I/O scheduler in Linux

BFQ is an accurate I/O scheduler, which makes non-trivial decisions.
This costs time, and, in practice, limits the maximum IOPS to only
about one million on a fast CPU. But Jens and the other core I/O
developers aim at (multi-)millions of IOPS. To reach such a high
speed, one needs an almost non-existent I/O scheduler, or no scheduler
at all. We argued that this speed is reached only by niche, high-end
enterprise hardware (most of which proprietary). So, by setting a
minimal I/O scheduler as default, to support this super fast hardware,
one severely affects the performance of 99+% of existing systems,
i.e., of every server and data-center node with rotational storage or
with 'normal' SSDs, as well as every personal system (from high-end
desktops to embedded systems). But we failed to convince Jens and
other core I/O developers to switch to BFQ instead, because their
only/main reference systems are those reaching millions of IOPS.

[1] https://algo.ing.unimo.it/people/paolo/BFQ/

> --Benjamin Gilbert
>

[Benjamin Gilbert]

On Wed, Jul 31, 2019 at 1:26 PM Paolo Valente <paolo....@linaro.org> wrote:

So, even if BFQ is available in the kernel, the burden to switch to
BFQ, from userspace, is left to either the user-space part of the
operating system, or the final users.  If your OS sticks to the current
default, then, as you can imagine, 99.99% of your users will naturally
interpret it as a sane default, and will stick to it too.

Hi Paolo,

Thanks for the additional info.  Container Linux is currently in maintenance mode and we're focusing our efforts on Fedora CoreOS.  We're unlikely to make this sort of configuration change to Container Linux at this point in its lifecycle.

Best,

--Benjamin Gilbert