kswapd stuck using 100% CPU

[Anton Blanchard]

Hi,

I booted the latest git today on a ppc64 box. When I pushed it into
swap I noticed both kswapd's were using 100% CPU and the soft lockup
detector suggested it was stuck in balance_pgdat:

BUG: soft lockup - CPU#7 stuck for 23s! [kswapd1:359]
Call Trace:
[c00000000015e190] .balance_pgdat+0x150/0x940
[c00000000015eb2c] .kswapd+0x1ac/0x490
[c00000000009edbc] .kthread+0xbc/0xd0
[c00000000002142c] .kernel_thread+0x54/0x70

I haven't had time to bisect but I did notice we were looping here:

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7658fd6..c92bad2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2945,9 +2959,11 @@ out:
if (zone->all_unreclaimable && priority != DEF_PRIORITY)
continue;

+#if 0
/* Would compaction fail due to lack of free memory? */
if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
goto loop_again;
+#endif

/* Confirm the zone is balanced for order-0 */
if (!zone_watermark_ok(zone, 0,


After commenting it out the box is happy again.

Anton
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Rik van Riel]

On Sat, 24 Mar 2012 13:03:53 +1100
Anton Blanchard <an...@samba.org> wrote:

> I booted the latest git today on a ppc64 box. When I pushed it into
> swap I noticed both kswapd's were using 100% CPU and the soft lockup
> detector suggested it was stuck in balance_pgdat:
>
> BUG: soft lockup - CPU#7 stuck for 23s! [kswapd1:359]
> Call Trace:
> [c00000000015e190] .balance_pgdat+0x150/0x940
> [c00000000015eb2c] .kswapd+0x1ac/0x490
> [c00000000009edbc] .kthread+0xbc/0xd0
> [c00000000002142c] .kernel_thread+0x54/0x70

Are you running without CONFIG_COMPACTION enabled by any chance?

Because if you do, the stub function compaction_suitable will always
return COMPACT_SKIPPED:

> I haven't had time to bisect but I did notice we were looping here:
>

> +++ b/mm/vmscan.c
> @@ -2945,9 +2959,11 @@ out:
> if (zone->all_unreclaimable && priority != DEF_PRIORITY)
> continue;
>
> +#if 0
> /* Would compaction fail due to lack of free memory? */
> if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
> goto loop_again;
> +#endif

The patch below should fix it.

-----

Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
otherwise the stub compaction_suitable function will always return
COMPACT_SKIPPED and send kswapd into an infinite loop.

Signed-off-by: Rik van Riel <ri...@redhat.com>
Reported-by: Anton Blanchard <an...@samba.org>

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7658fd6..33c332b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2946,7 +2946,8 @@ out:
continue;

/* Would compaction fail due to lack of free memory? */

- if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
+ if (COMPACTION_BUILD &&
+ compaction_suitable(zone, order) == COMPACT_SKIPPED)
goto loop_again;

/* Confirm the zone is balanced for order-0 */

[Hugh Dickins]

On Sat, 24 Mar 2012, Rik van Riel wrote:
>
> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> otherwise the stub compaction_suitable function will always return
> COMPACT_SKIPPED and send kswapd into an infinite loop.
>
> Signed-off-by: Rik van Riel <ri...@redhat.com>
> Reported-by: Anton Blanchard <an...@samba.org>

Thank you, Anton and Rik. I never quite got around to investigating
why swapping had been nearly twice as slow with linux-next on my Aspire
One (with a relatively minimal config, omitting COMPACTION). That was
the reason (one half of the HT cpu busy in kswapd), and this fixes it.

Tested-by: Hugh Dickins <hu...@google.com>

[Mel Gorman]

On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
>
> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> otherwise the stub compaction_suitable function will always return
> COMPACT_SKIPPED and send kswapd into an infinite loop.
>
> Signed-off-by: Rik van Riel <ri...@redhat.com>
> Reported-by: Anton Blanchard <an...@samba.org>
>

Acked-by: Mel Gorman <m...@csn.ul.ie>

--
Mel Gorman
SUSE Labs

[Pekka Enberg]

On Mon, Mar 26, 2012 at 12:32 PM, Mel Gorman <m...@csn.ul.ie> wrote:
> On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
>>
>> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
>> otherwise the stub compaction_suitable function will always return
>> COMPACT_SKIPPED and send kswapd into an infinite loop.
>>
>> Signed-off-by: Rik van Riel <ri...@redhat.com>
>> Reported-by: Anton Blanchard <an...@samba.org>
>
> Acked-by: Mel Gorman <m...@csn.ul.ie>

The API looks fragile and this patch isn't exactly making it any
better. Why don't we make compaction_suitable() return something other
than COMPACT_SKIPPED for !CONFIG_COMPACTION case?

Pekka

[Mel Gorman]

On Mon, Mar 26, 2012 at 01:40:41PM +0300, Pekka Enberg wrote:
> On Mon, Mar 26, 2012 at 12:32 PM, Mel Gorman <m...@csn.ul.ie> wrote:
> > On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
> >>
> >> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> >> otherwise the stub compaction_suitable function will always return
> >> COMPACT_SKIPPED and send kswapd into an infinite loop.
> >>
> >> Signed-off-by: Rik van Riel <ri...@redhat.com>
> >> Reported-by: Anton Blanchard <an...@samba.org>
> >
> > Acked-by: Mel Gorman <m...@csn.ul.ie>
>
> The API looks fragile and this patch isn't exactly making it any
> better. Why don't we make compaction_suitable() return something other
> than COMPACT_SKIPPED for !CONFIG_COMPACTION case?
>

Returning COMPACT_PARTIAL or COMPACT_CONTINUE would confuse the check in
should_continue_reclaim. A fourth return type could be added but an
obvious name does not spring to mind that would end up being similar to
just adding a CONFIG_COMPACTION check.

--
Mel Gorman
SUSE Labs

[Pekka Enberg]

Hi Mel,

On Mon, Mar 26, 2012 at 4:56 PM, Mel Gorman <m...@csn.ul.ie> wrote:
>> The API looks fragile and this patch isn't exactly making it any
>> better. Why don't we make compaction_suitable() return something other
>> than COMPACT_SKIPPED for !CONFIG_COMPACTION case?
>
> Returning COMPACT_PARTIAL or COMPACT_CONTINUE would confuse the check in
> should_continue_reclaim. A fourth return type could be added but an
> obvious name does not spring to mind that would end up being similar to
> just adding a CONFIG_COMPACTION check.

How about COMPACT_DISABLED?

The current API just doesn't make sense from practical point of view.
Anyone calling compaction_suitable() needs to do the COMPAT_BUILD
check first which is a non-obvious and error-prone API.

Pekka