bfq v7r8 on Linux 3.10 very slow on Android

[Ju Hyung Park]

v7r6 tested by Paolo, against CFQ, DEADLINE and NOOP : https://www.youtube.com/watch?v=1cjZeaCXIyM

I knew that BFQ is a strong custom I/O scheduler for Linux, but I was even more excited to try it out.

But with my own testing of v7r8(the latest when I was testing it), I cannot be more disappointed.

https://www.youtube.com/watch?v=dSknEcbxg74

During background writes, BFQ was 3 times slower than ROW(Read-Over-Write by Qualcomm : https://www.codeaurora.org/cgit/quic/la/kernel/msm-3.10/log/block/row-iosched.c?h=LA.BR.1.3.2 ).

Is BFQ weak against background writes?

And during idle background, BFQ was still noticeably slower than ROW, CFQ and FIOPS(which is known to produce best sequential read/write numbers for flash).

While my testing methodology may not be entirely scientific, it's clear to me that BFQ suffers.

Any ideas why my testings has gone that way?

Or should I tweak some tunables from sysfs?

Or is something like https://github.com/CyanogenMod/android_system_core/commit/f097d5dff82fa4918a3731576a23ddb0ec7fd932 and https://github.com/CyanogenMod/android_vendor_cm/blob/cm-13.0/prebuilt/common/etc/init.local.rc (~ Line 25) required?

[Paolo Valente]

Hi,
thanks for testing BFQ, and congratulations for your very nice (and, in my opinion, professional) video.

Unfortunately, your results are not very surprising for me. As I had the opportunity to say also in this group, I have defined the main heuristics of BFQ for PCs. In fact, the video of mine that you mention does show results on a PC. Then, several people wondered: “Hey, if BFQ is so good on a PC, why not to use it on an Android system? In the end, PC and smartphone systems are becoming more and more similar”.

Well, they are not yet similar enough. So, on some devices, especially low and med-level ones, BFQ happens to be better than the other schedulers, on others it does not. But on all devices, all schedulers do not perform well, and this is the actual problem. As your tests show, even the best scheduler suffers from about a 100% increase of latencies with just one writer. You may get much worse results by adding also one reader (which would mimic more closely what happens in a modern Android device, during unlucky time intervals).

So, this is the problem. Next interesting questions may be: what are the causes? are there solutions?

Here comes the first surprise (at least it was a surprise for me when we discovered it): the causes have nothing to do with the schedulers! The causes just interact in different ways with different schedulers, but the increase in latency is not the schedulers' fault. It is a blocking problem occurring inside the kernel, and due to virtual-memory handling and journaling.

Are there solutions? Yes. And we have already developed a first set of these solutions. They have still a long way to go before becoming production solutions. I am working on these solutions with my brave students, plus other volunteer collaborators. This work is now also supported by Linaro. As of now, I can only anticipate: with your test, and after applying our solution, the start-up time with and without writes would have been the same. I hope we will able soon to share at least some initial prototype.

Sorry for the long response. If you have further questions, do not hesitate to ask.

Paolo

> --
> You received this message because you are subscribed to the Google Groups "bfq-iosched" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bfq-iosched...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ju Hyung Park]

Hi.

Thanks for the detailed answer!

Can I ask what's the name of that project?

I'm currently interested in EAS(Energy-Aware-Scheduler) which seems to be integrating CPU governor with CFS scheduler tightly

for not only better energy consumption, but also faster tasks switching and general performance.

Also, I want to ask one more thing.

The Qualcomm-made ROW scheduler, which seems to always prioritize reads over writes, seems to generally work great on almost all Android devices(maybe because of its mobile characteristic).

If we'd use ROW on PCs, what would be its weaknesses?

2016년 1월 22일 금요일 오후 5시 57분 9초 UTC+9, paolo 님의 말:

[Paolo Valente]

Il giorno 22/gen/2016, alle ore 14:30, Ju Hyung Park <qkrwn...@gmail.com> ha scritto:

> Hi.
> Thanks for the detailed answer!
>
> Can I ask what's the name of that project?

Good question. I just realized that I never gave it a proper name, or at least an acronym!

You can find a short description of the project here:

http://algogroup.unimore.it/algodev/ideas-list.php

after the title "Extending BFQ benefits to Android systems".

Should you want to contribute, you are of course welcome.

> I'm currently interested in EAS(Energy-Aware-Scheduler) which seems to be integrating CPU governor with CFS scheduler tightly
> for not only better energy consumption, but also faster tasks switching and general performance.
>

I know that very interesting project. I have done something also for CPU scheduling, but this is really off topic.

> Also, I want to ask one more thing.
> The Qualcomm-made ROW scheduler, which seems to always prioritize reads over writes, seems to generally work great on almost all Android devices(maybe because of its mobile characteristic).
> If we'd use ROW on PCs, what would be its weaknesses?

Exactly its strength, I’m afraid: it prioritizes reads over writes :)

Some applications rely on write completions to go on, especially in Android, where there is a maniacal concern in guaranteeing that every written datum is stored in persistent memory as soon as possible. A trivial example is the telephone app, which most likely issues a potentially blocking write-and-flush when someone calls you, to log the new incoming call.

Should you want to try:
1) launch at least one reader and one writer in parallel
2) kill the telephone app (this step is FUNDAMENTAL, otherwise the next one is perfectly useless for this test)
3) drop caches
4) ask someone to call you :)

However, I have never tried this myself so far, so mine are just speculations … (currently, almost all my little developer time is absorbed by the BFQ project)

Thanks,
Paolo

[Ju Hyung Park]

That would be an interesting test.

I'll do it when I have the time :)

And it seems like BFQ team does acknowledge that CM uses BFQ on many Android devices.

Considering my tests and your reply : "Unfortunately, your results are not very surprising for me",

why would CM use BFQ from the first place when :

A) BFQ not yet being fully ready for Android and its characteristics

B) ROW(which is the default on Snapdragon devices, and which is like 99% of CM's) still kicking ass on many places ?

2016년 1월 22일 금요일 오후 11시 10분 25초 UTC+9, paolo 님의 말:

[Paolo Valente]

Il 22/01/2016 15:23, Ju Hyung Park ha scritto:

That would be an interesting test.

I'll do it when I have the time :)

And it seems like BFQ team does acknowledge that CM uses BFQ on many Android devices.

Considering my tests and your reply : "Unfortunately, your results are not very surprising for me",

why would CM use BFQ from the first place when :

A) BFQ not yet being fully ready for Android and its characteristics

B) ROW(which is the default on Snapdragon devices, and which is like 99% of CM's) still kicking ass on many places ?

Unfortunately I am not in touch with the developers that have evaluated BFQ for CM, so I can only guess that, as it happened to me when I tried a few years ago, they have seen good results on some devices. But, honestly, my only good reply to your question is going on with our truly effective solution to the problem of a high latency in the presence of background I/O.