On Sat, 13 Aug 2016, Ritesh Raj Sarraf wrote:

> Hi,
>
> As per the description, BFQ is touted as a low-latency scheduler for
> interactivity. 
[...]
> The above description covers reads as well as *writes*. But on some of the demo
> videos I saw online, IIRC it only demonstrated about the read scenarios.
>
> Hence, I'm writing here with some details about my write use case. It sure is an
> exceptional case, but I think it shouldn't qualify as a corner case.
[...]
> With just a single I/O thread of dd, I have been able to always bring the
> machine to a hung state. The only clean exit is to let it be as is, and the I/O
> thread to complete. Upon completion, the kernel is capable of recovering.
>
> Point to keep in mind, in this case, is:
>
> * Single I/O thread
> * Ensure that you write data more than your full RAM capacity
>
> Command run: dd if=/dev/zero of=/tmp/foo.img bs=1M count=10000 conv=fsync;
>
> With the above command running, as soon as the memory gets full, the OS goes
> into a hung state. It does recover but I'm sure I can make it thrash further if
> I increase the number of processes contending for I/O.
>
> Note: The catch is to pump total I/O, which is more than the amount of physical
> RAM you have. So, in above example, I chose 10000 MiB because I have around 8
> GiB of physical RAM.
>
> Attached is the dmesg log, where the kernel went haywire, once the I/O and
> Memory pressure increased.
>
> I was hoping that BFQ was capable of improving such scenarios. But I was able to
> make the kernel stall, maybe not as bad as CFQ, but still it stalled. IIRC, cfq
> used to completely thrash for hours, under this test.

I think there are two issues going on here.  One is a VMM issue because
you have exhausted your machine to the point that your wifi driver [wl]
can't allocate even a single memory page for its use (order 0).  To solve
that problem, try this:

echo 262144 > /proc/sys/vm/min_free_kbytes

The second issue you may not have measured, as it pertains the IO latency
of dd (which you might not care about and dd doesn't measure). The
networking group would call the issue issue "buffer bloat".

In your scenario, you have 8GB of buffers.  You don't need that much
buffer to exhaust your IO bandwidth and keep the pipe full, but the kernel
VFS/VMM pagecache is willing to accommodate.  To solve that, create a
memory cgroup, say 512mb, and run `dd` under that cgroup (or run dd with
whatever flag sets O_DIRECT to completely bypass the pagecache).

The cgroup would give ~512mb of buffer to dd which should be plenty for a
linear write without excessive pagecache eviction; your remaining 7.5GB
could then be useful for other application page caches (filesystem and
shared library mmaps).  Also, 512mb reduces the per-IO completion latency
of dd's IO path from $(8192mb/disk-bandwidth)$ to
$(512mb/disk-bandwidth)$.  If disk-bandwidth is 128mb/sec, then you have
queue latency of 64s in the former case, and 4s in the latter.  Imagine, a
64s ping!

While targeted at the network stack, this article provides a wonderful
discussion of queue-size latency considerations:
        http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:networking:traffic_control

In Linux net/ parlance, BFQ would be called a queuing discipline
(qdisc); the most similar queuing discipline to BFQ is probably HTB -
Hierarchical Token Bucket.  A disk or net queue is still a queue, so while
disk and network queues have different fairness and implementation
considerations, the fundamental concepts are similar and both can be tuned
for latency vs bandwidth.

--
Eric Wheeler

>
>
> So, is this test valid ? If so, can BFQ be improved to tackle this scenario?
> Maybe other BFQ users could try the above scenario and share their results. Keep
> in mind that the issue is quickly reproducible on rotational backing device, but
> I've also been able to reproduce it on SSDs.
>
>
> --
> Ritesh Raj Sarraf
> RESEARCHUT - http://www.researchut.com
> "Necessity is the mother of invention."
>
> --
> You received this message because you are subscribed to the Google Groups "bfq-iosched" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bfq-i...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>