swap/page problem

[Kevin Oberman]

During a large build (llvm13), my system ground to a near halt with almost everything suspended. After several minutes, the system slowly recovered. When I looked at the messages log, I found 57 kernel messages spread over 3.75 hours, in the form of:

wap_pager: indefinite wait buffer: bufobj: 0, blkno: 862845, size: 20480

The block numbers and sizes varied. bufobj was always '0'. I had significant swap available swap space,as far as I could tell. I have 20GB of RAM and 24GB of swap. I am running stable 48937-3c6b6246f2f from January 13.

I know that the LLVM build is huge, but I've not seen this before. What, exactly, is this message telling me? Am I out of RAM and swap? I couldd add another 24GB of swap, though it would be on spinning rust, not SSD.

Thanks!

--

Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkob...@gmail.com

PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683

[tech-lists]

Hi,

On Thu, Feb 10, 2022 at 02:28:50PM -0800, Kevin Oberman wrote:
>During a large build (llvm13), my system ground to a near halt with almost
>everything suspended. After several minutes, the system slowly recovered.
>When I looked at the messages log, I found 57 kernel messages spread over
>3.75 hours, in the form of:
>wap_pager: indefinite wait buffer: bufobj: 0, blkno: 862845, size: 20480
>
>The block numbers and sizes varied. bufobj was always '0'. I had
>significant swap available swap space,as far as I could tell. I have 20GB
>of RAM and 24GB of swap. I am running stable 48937-3c6b6246f2f from January
>13.
>
>I know that the LLVM build is huge, but I've not seen this before. What,
>exactly, is this message telling me? Am I out of RAM and swap? I couldd add
>another 24GB of swap, though it would be on spinning rust, not SSD.

I've seen this problem before and came to the conclusion
after reading threads on the lists and asking questions that
it wasn't a swap problem as primary issue but maybe theres some contention
between processes which affects llvm in particular and makes it eat swap.
Also, as you noted, swap doesn't really run out.

I use poudriere to build and have parallel jobs set to 1 now
with make jobs enabled and the problem doesn't happen. If you're just
using the ports tree in the traditional way, try make -j1.
If parallel jobs is unset it would use hw.ncpu which here is 8 which would
produce the problem you descrive when compiling llvm13.

These sysctls are now set:

vfs.read_max=128 # default 64 - speeds up disk i/o
vfs.aio.max_buf_aio=8192
vfs.aio.max_aio_queue_per_proc=65536
vfs.aio.max_aio_per_proc=8192
vfs.aio.max_aio_queue=65536
vm.pageout_oom_seq=120
vm.pfault_oom_attempts=-1

those last two may be especially helpful for your situation.

My context here is amd64 i7-4770K (so 8 cpus with HT) clocked to 4.3GHz
and the disk used for building poers is SSD wiht 16GB swap (partition).
RAM is 32GB

--
J.

[Kevin Oberman]

Thanks for the suggestions. The AIO ones look like those I recommended for vbox, though they are no longer needed for that as vbox has been modified to no longer use AIO. In any case, I'll see what happens.

One oddity is that the problem seems to occur when my system reports "critical temperature detected". Shortly after, I see the swap_pager messages start. I have concluded that there is absolutely no issue with either RAM or swap space. Whatever is happening, it is tied to the problems that I have been seeing since I got my Lenovo L-13: P-States disabled, weird CPU frequency behavior, strange thermal control issues. I'm really regretting getting this laptop.

Thanks again for the suggestions!