[PATCH] mm: add free_hot_cold_page_list helper

[Konstantin Khlebnikov]

This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
It frees pages directly from list without temporary page-vector.
It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.

bloat-o-meter:

add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
function old new delta
free_hot_cold_page_list - 264 +264
get_page_from_freelist 2129 2132 +3
__pagevec_free 243 239 -4
split_free_page 380 373 -7
release_pages 606 510 -96
free_page_list 188 - -188

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---
include/linux/gfp.h | 1 +
mm/page_alloc.c | 12 ++++++++++++
mm/swap.c | 14 +++-----------
mm/vmscan.c | 20 +-------------------
4 files changed, 17 insertions(+), 30 deletions(-)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index cb40892..dd7b9cc 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -358,6 +358,7 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
extern void __free_pages(struct page *page, unsigned int order);
extern void free_pages(unsigned long addr, unsigned int order);
extern void free_hot_cold_page(struct page *page, int cold);
+extern void free_hot_cold_page_list(struct list_head *list, int cold);

#define __free_page(page) __free_pages((page), 0)
#define free_page(addr) free_pages((addr), 0)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1dbcf88..af486e4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1209,6 +1209,18 @@ out:
local_irq_restore(flags);
}

+void free_hot_cold_page_list(struct list_head *list, int cold)
+{
+ struct page *page, *next;
+
+ list_for_each_entry_safe(page, next, list, lru) {
+ trace_mm_pagevec_free(page, cold);
+ free_hot_cold_page(page, cold);
+ }
+
+ INIT_LIST_HEAD(list);
+}
+
/*
* split_page takes a non-compound higher-order page, and splits it into
* n (1<<order) sub-pages: page[0..n]
diff --git a/mm/swap.c b/mm/swap.c
index 3a442f1..b9138c7 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -562,11 +562,10 @@ int lru_add_drain_all(void)
void release_pages(struct page **pages, int nr, int cold)
{
int i;
- struct pagevec pages_to_free;
+ LIST_HEAD(pages_to_free);
struct zone *zone = NULL;
unsigned long uninitialized_var(flags);

- pagevec_init(&pages_to_free, cold);
for (i = 0; i < nr; i++) {
struct page *page = pages[i];

@@ -597,19 +596,12 @@ void release_pages(struct page **pages, int nr, int cold)
del_page_from_lru(zone, page);
}

- if (!pagevec_add(&pages_to_free, page)) {
- if (zone) {
- spin_unlock_irqrestore(&zone->lru_lock, flags);
- zone = NULL;
- }
- __pagevec_free(&pages_to_free);
- pagevec_reinit(&pages_to_free);
- }
+ list_add_tail(&page->lru, &pages_to_free);
}
if (zone)
spin_unlock_irqrestore(&zone->lru_lock, flags);

- pagevec_free(&pages_to_free);
+ free_hot_cold_page_list(&pages_to_free, cold);
}
EXPORT_SYMBOL(release_pages);

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7ef6912..47403c9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -737,24 +737,6 @@ static enum page_references page_check_references(struct page *page,
return PAGEREF_RECLAIM;
}

-static noinline_for_stack void free_page_list(struct list_head *free_pages)
-{
- struct pagevec freed_pvec;
- struct page *page, *tmp;
-
- pagevec_init(&freed_pvec, 1);
-
- list_for_each_entry_safe(page, tmp, free_pages, lru) {
- list_del(&page->lru);
- if (!pagevec_add(&freed_pvec, page)) {
- __pagevec_free(&freed_pvec);
- pagevec_reinit(&freed_pvec);
- }
- }
-
- pagevec_free(&freed_pvec);
-}
-
/*
* shrink_page_list() returns the number of reclaimed pages
*/
@@ -996,7 +978,7 @@ keep_lumpy:
if (nr_dirty && nr_dirty == nr_congested && scanning_global_lru(sc))
zone_set_flag(zone, ZONE_CONGESTED);

- free_page_list(&free_pages);
+ free_hot_cold_page_list(&free_pages, 1);

list_splice(&ret_pages, page_list);
count_vm_events(PGACTIVATE, pgactivate);

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Andrew Morton]

On Fri, 29 Jul 2011 11:58:37 +0400
Konstantin Khlebnikov <khleb...@openvz.org> wrote:

> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> It frees pages directly from list without temporary page-vector.
> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
>
> bloat-o-meter:
>
> add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
> function old new delta
> free_hot_cold_page_list - 264 +264
> get_page_from_freelist 2129 2132 +3
> __pagevec_free 243 239 -4
> split_free_page 380 373 -7
> release_pages 606 510 -96
> free_page_list 188 - -188
>

It saves a total of 150 bytes for me.

There's a potential problem here with cache longevity. If
release_pages() is called with a large number of pages then the current
code's approach of freeing pages 16-at-a-time will hopefully cause
those pageframes to still be in CPU cache when we get to actually
freeing them.

But after this change, we free all the pages in a single operation
right at the end, which adds risk that we'll have to reload all their
pageframes into CPU cache again.

That'll only be a problem if release_pages() _is_ called with a large
number of pages. And manipulating large numbers of pages represents a
lot of work, so the additional work from one cachemiss per page will
presumably be tiny.

[Konstantin Khlebnikov]

all release_pages() callers (except fuse) call it for pages array not bigger than PAGEVEC_SIZE (=14).
while for fuse it put likely not last page reference, so we didn't free them on this path.

[Minchan Kim]

I understand you want to minimize changes without breaking current ABI
with trace tools.
But apparently, It's not a pagvec_free. It just hurts readability.
As I take a look at the code, mm_pagevec_free isn't related to pagevec
but I guess it can represent 0-order pages free because 0-order pages
are freed only by pagevec until now.
So, how about renaming it with mm_page_free or mm_page_free_zero_order?
If you do, you need to do s/MM_PAGEVEC_FREE/MM_FREE_FREE/g in
trace-pagealloc-postprocess.pl.

> +               free_hot_cold_page(page, cold);
> +       }
> +
> +       INIT_LIST_HEAD(list);

Why do we need it?

--
Kind regards,
Minchan Kim

[Andrew Morton]

On Mon, 29 Aug 2011 16:48:46 +0900
Minchan Kim <minch...@gmail.com> wrote:

> On Fri, Jul 29, 2011 at 4:58 PM, Konstantin Khlebnikov
> <khleb...@openvz.org> wrote:
> > This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> > It frees pages directly from list without temporary page-vector.
> > It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
> >
> > bloat-o-meter:
> >
> > add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
> > function old new delta
> > free_hot_cold_page_list - 264 +264
> > get_page_from_freelist 2129 2132 +3

> > pagevec_free 243 239 -4
> > split_free_page 380 373 -7
> > release_pages 606 510 -96
> > free_page_list 188 - -188
> >
> > Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
> > ---
> > include/linux/gfp.h | 1 +
> > mm/page_alloc.c | 12 ++++++++++++
> > mm/swap.c | 14 +++-----------
> > mm/vmscan.c | 20 +-------------------
> > 4 files changed, 17 insertions(+), 30 deletions(-)
> >
> > diff --git a/include/linux/gfp.h b/include/linux/gfp.h
> > index cb40892..dd7b9cc 100644
> > --- a/include/linux/gfp.h
> > +++ b/include/linux/gfp.h
> > @@ -358,6 +358,7 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);

> > extern void free_pages(struct page *page, unsigned int order);

> > extern void free_pages(unsigned long addr, unsigned int order);
> > extern void free_hot_cold_page(struct page *page, int cold);
> > +extern void free_hot_cold_page_list(struct list_head *list, int cold);
> >

> > #define free_page(page) free_pages((page), 0)

My email has been horrid for a couple of months (fixed now), so I might
have missed any reply to Minchin's review comments?

[Konstantin Khlebnikov]

This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.

It frees pages directly from the list without temporary page-vector.

It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.

bloat-o-meter:

add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
function old new delta
free_hot_cold_page_list - 264 +264
get_page_from_freelist 2129 2132 +3
__pagevec_free 243 239 -4
split_free_page 380 373 -7
release_pages 606 510 -96
free_page_list 188 - -188

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---
include/linux/gfp.h | 1 +

mm/page_alloc.c | 13 +++++++++++++

mm/swap.c | 14 +++-----------
mm/vmscan.c | 20 +-------------------

4 files changed, 18 insertions(+), 30 deletions(-)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 3a76faf..6562958 100644

--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -358,6 +358,7 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
extern void __free_pages(struct page *page, unsigned int order);
extern void free_pages(unsigned long addr, unsigned int order);
extern void free_hot_cold_page(struct page *page, int cold);
+extern void free_hot_cold_page_list(struct list_head *list, int cold);

#define __free_page(page) __free_pages((page), 0)
#define free_page(addr) free_pages((addr), 0)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c

index 9dd443d..5093114 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1211,6 +1211,19 @@ out:
}

/*
+ * Free a list of 0-order pages
+ */

+void free_hot_cold_page_list(struct list_head *list, int cold)
+{
+ struct page *page, *next;
+
+ list_for_each_entry_safe(page, next, list, lru) {
+ trace_mm_pagevec_free(page, cold);
+ free_hot_cold_page(page, cold);
+ }
+}
+

+/*

* split_page takes a non-compound higher-order page, and splits it into
* n (1<<order) sub-pages: page[0..n]

* Each sub-page must be freed individually.

index a90c603..77f84ef 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -728,24 +728,6 @@ static enum page_references page_check_references(struct page *page,

return PAGEREF_RECLAIM;
}

-static noinline_for_stack void free_page_list(struct list_head *free_pages)
-{
- struct pagevec freed_pvec;
- struct page *page, *tmp;
-
- pagevec_init(&freed_pvec, 1);
-
- list_for_each_entry_safe(page, tmp, free_pages, lru) {
- list_del(&page->lru);
- if (!pagevec_add(&freed_pvec, page)) {
- __pagevec_free(&freed_pvec);
- pagevec_reinit(&freed_pvec);
- }
- }
-
- pagevec_free(&freed_pvec);
-}
-
/*
* shrink_page_list() returns the number of reclaimed pages
*/

@@ -1009,7 +991,7 @@ keep_lumpy:

[Konstantin Khlebnikov]

Sorry, I forget about this patch. v2 sended.

[Andrew Morton]

On Tue, 01 Nov 2011 11:45:02 +0300
Konstantin Khlebnikov <khleb...@openvz.org> wrote:

> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> It frees pages directly from the list without temporary page-vector.
> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
>
> bloat-o-meter:
>
> add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
> function old new delta
> free_hot_cold_page_list - 264 +264
> get_page_from_freelist 2129 2132 +3
> __pagevec_free 243 239 -4
> split_free_page 380 373 -7
> release_pages 606 510 -96
> free_page_list 188 - -188

Here's what you changed:

--- a/mm/page_alloc.c~mm-add-free_hot_cold_page_list-helper-v2
+++ a/mm/page_alloc.c
@@ -1210,6 +1210,9 @@ out:
local_irq_restore(flags);
}

+/*

+ * Free a list of 0-order pages
+ */

void free_hot_cold_page_list(struct list_head *list, int cold)

{
struct page *page, *next;
@@ -1218,8 +1221,6 @@ void free_hot_cold_page_list(struct list
trace_mm_pagevec_free(page, cold);
free_hot_cold_page(page, cold);
}
-
- INIT_LIST_HEAD(list);
}

/*
_

However I can't find any sign that you addressed Minchin's original
review comment regarding free_hot_cold_page_list():

: I understand you want to minimize changes without breaking current ABI

: with trace tools.
: But apparently, It's not a pagvec_free. It just hurts readability.
: As I take a look at the code, mm_pagevec_free isn't related to pagevec
: but I guess it can represent 0-order pages free because 0-order pages
: are freed only by pagevec until now.
: So, how about renaming it with mm_page_free or mm_page_free_zero_order?
: If you do, you need to do s/MM_PAGEVEC_FREE/MM_FREE_FREE/g in
: trace-pagealloc-postprocess.pl.

[Hugh Dickins]

On Tue, 1 Nov 2011, Konstantin Khlebnikov wrote:

> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> It frees pages directly from the list without temporary page-vector.
> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.

Sorry for not speaking up sooner, but I do like this patch very much
(and I'm content with your trace compatibility choice - whatever).

Not so much in itself, but because it then allows a further patch
(mainly to mm/vmscan.c) to remove two levels of pagevec, reducing
its deepest stack by around 240 bytes.

I have that patch, but keep putting off sending it in, because I want
to show a reclaim stack overflow that it prevents, but the new avoidance
of writeback in direct reclaim makes that harder to demonstrate. Damn!

One question on your patch: where you have release_pages() doing
> + list_add_tail(&page->lru, &pages_to_free);

That seems reasonable, but given that __pagevec_free() proceeds by
while (--i >= 0) {
, starting from the far end of the pagevec (the most recently added
struct page, the most likely to be hot), wouldn't you reproduce
existing behaviour more accurately by a simple list_add()?

Or have I got that back to front? If so, a comment on the
list_add_tail() would help me to remember why - thanks.

Hugh

[Konstantin Khlebnikov]

Now I try to handle this, but this events logic little bit messy:
kernel always emit mm_page_free_direct event even if we free it in batch.
We really need two events for page-free? Maybe we can just remove pagevec-free and
rename page_free_direct into page_free?

BTW, pagevec_free now absolutely unused, I'll remove it in separate patch.

[Konstantin Khlebnikov]

Hugh Dickins wrote:
> On Tue, 1 Nov 2011, Konstantin Khlebnikov wrote:
>
>> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
>> It frees pages directly from the list without temporary page-vector.
>> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
>
> Sorry for not speaking up sooner, but I do like this patch very much
> (and I'm content with your trace compatibility choice - whatever).
>
> Not so much in itself, but because it then allows a further patch
> (mainly to mm/vmscan.c) to remove two levels of pagevec, reducing
> its deepest stack by around 240 bytes.

That's Great. Also I don't like pagevec-based free because we sometimes do
extra lru lock-unlock on vector overflow.

>
> I have that patch, but keep putting off sending it in, because I want
> to show a reclaim stack overflow that it prevents, but the new avoidance
> of writeback in direct reclaim makes that harder to demonstrate. Damn!
>
> One question on your patch: where you have release_pages() doing
>> + list_add_tail(&page->lru,&pages_to_free);
>
> That seems reasonable, but given that __pagevec_free() proceeds by
> while (--i>= 0) {
> , starting from the far end of the pagevec (the most recently added
> struct page, the most likely to be hot), wouldn't you reproduce
> existing behaviour more accurately by a simple list_add()?
>
> Or have I got that back to front? If so, a comment on the
> list_add_tail() would help me to remember why - thanks.
>
> Hugh

Ok, this reasonable. Any way, the second its user: shrink_page_list() puts pages at the front.

[Konstantin Khlebnikov]

Rename mm_page_free_direct into mm_page_free
and mm_pagevec_free into mm_page_free_batched

Since v2.6.33-5426-gc475dab kernel trigger mm_page_free_direct for all freed pages,
not only for directly freed. So, let's name it properly.
For pages freed via page-list we also trigger mm_page_free_batched event.

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---

Documentation/trace/events-kmem.txt | 12 ++++++------
.../postprocess/trace-pagealloc-postprocess.pl | 20 ++++++++++----------
include/trace/events/kmem.h | 4 ++--
mm/page_alloc.c | 4 ++--
4 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/Documentation/trace/events-kmem.txt b/Documentation/trace/events-kmem.txt
index aa82ee4..1948004 100644
--- a/Documentation/trace/events-kmem.txt
+++ b/Documentation/trace/events-kmem.txt
@@ -40,8 +40,8 @@ but the call_site can usually be used to extrapolate that information.
==================
mm_page_alloc page=%p pfn=%lu order=%d migratetype=%d gfp_flags=%s
mm_page_alloc_zone_locked page=%p pfn=%lu order=%u migratetype=%d cpu=%d percpu_refill=%d
-mm_page_free_direct page=%p pfn=%lu order=%d
-mm_pagevec_free page=%p pfn=%lu order=%d cold=%d
+mm_page_free page=%p pfn=%lu order=%d
+mm_page_free_batched page=%p pfn=%lu order=%d cold=%d

These four events deal with page allocation and freeing. mm_page_alloc is
a simple indicator of page allocator activity. Pages may be allocated from
@@ -53,13 +53,13 @@ amounts of activity imply high activity on the zone->lock. Taking this lock
impairs performance by disabling interrupts, dirtying cache lines between
CPUs and serialising many CPUs.

-When a page is freed directly by the caller, the mm_page_free_direct event
+When a page is freed directly by the caller, the only mm_page_free event
is triggered. Significant amounts of activity here could indicate that the
callers should be batching their activities.

-When pages are freed using a pagevec, the mm_pagevec_free is
-triggered. Broadly speaking, pages are taken off the LRU lock in bulk and
-freed in batch with a pagevec. Significant amounts of activity here could
+When pages are freed in batch, the also mm_page_free_batched is triggered.
+Broadly speaking, pages are taken off the LRU lock in bulk and
+freed in batch with a page list. Significant amounts of activity here could
indicate that the system is under memory pressure and can also indicate
contention on the zone->lru_lock.

diff --git a/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl b/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl
index 7df50e8..0a120aa 100644
--- a/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl
+++ b/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl
@@ -17,8 +17,8 @@ use Getopt::Long;

# Tracepoint events
use constant MM_PAGE_ALLOC => 1;
-use constant MM_PAGE_FREE_DIRECT => 2;
-use constant MM_PAGEVEC_FREE => 3;
+use constant MM_PAGE_FREE => 2;
+use constant MM_PAGE_FREE_BATCHED => 3;
use constant MM_PAGE_PCPU_DRAIN => 4;
use constant MM_PAGE_ALLOC_ZONE_LOCKED => 5;
use constant MM_PAGE_ALLOC_EXTFRAG => 6;
@@ -223,10 +223,10 @@ EVENT_PROCESS:
# Perl Switch() sucks majorly
if ($tracepoint eq "mm_page_alloc") {
$perprocesspid{$process_pid}->{MM_PAGE_ALLOC}++;
- } elsif ($tracepoint eq "mm_page_free_direct") {
- $perprocesspid{$process_pid}->{MM_PAGE_FREE_DIRECT}++;
- } elsif ($tracepoint eq "mm_pagevec_free") {
- $perprocesspid{$process_pid}->{MM_PAGEVEC_FREE}++;
+ } elsif ($tracepoint eq "mm_page_free") {
+ $perprocesspid{$process_pid}->{MM_PAGE_FREE}++
+ } elsif ($tracepoint eq "mm_page_free_batched") {
+ $perprocesspid{$process_pid}->{MM_PAGE_FREE_BATCHED}++;
} elsif ($tracepoint eq "mm_page_pcpu_drain") {
$perprocesspid{$process_pid}->{MM_PAGE_PCPU_DRAIN}++;
$perprocesspid{$process_pid}->{STATE_PCPU_PAGES_DRAINED}++;
@@ -336,8 +336,8 @@ sub dump_stats {
$process_pid,
$stats{$process_pid}->{MM_PAGE_ALLOC},
$stats{$process_pid}->{MM_PAGE_ALLOC_ZONE_LOCKED},
- $stats{$process_pid}->{MM_PAGE_FREE_DIRECT},
- $stats{$process_pid}->{MM_PAGEVEC_FREE},
+ $stats{$process_pid}->{MM_PAGE_FREE},
+ $stats{$process_pid}->{MM_PAGE_FREE_BATCHED},
$stats{$process_pid}->{MM_PAGE_PCPU_DRAIN},
$stats{$process_pid}->{HIGH_PCPU_DRAINS},
$stats{$process_pid}->{HIGH_PCPU_REFILLS},
@@ -364,8 +364,8 @@ sub aggregate_perprocesspid() {

$perprocess{$process}->{MM_PAGE_ALLOC} += $perprocesspid{$process_pid}->{MM_PAGE_ALLOC};
$perprocess{$process}->{MM_PAGE_ALLOC_ZONE_LOCKED} += $perprocesspid{$process_pid}->{MM_PAGE_ALLOC_ZONE_LOCKED};
- $perprocess{$process}->{MM_PAGE_FREE_DIRECT} += $perprocesspid{$process_pid}->{MM_PAGE_FREE_DIRECT};
- $perprocess{$process}->{MM_PAGEVEC_FREE} += $perprocesspid{$process_pid}->{MM_PAGEVEC_FREE};
+ $perprocess{$process}->{MM_PAGE_FREE} += $perprocesspid{$process_pid}->{MM_PAGE_FREE};
+ $perprocess{$process}->{MM_PAGE_FREE_BATCHED} += $perprocesspid{$process_pid}->{MM_PAGE_FREE_BATCHED};
$perprocess{$process}->{MM_PAGE_PCPU_DRAIN} += $perprocesspid{$process_pid}->{MM_PAGE_PCPU_DRAIN};
$perprocess{$process}->{HIGH_PCPU_DRAINS} += $perprocesspid{$process_pid}->{HIGH_PCPU_DRAINS};
$perprocess{$process}->{HIGH_PCPU_REFILLS} += $perprocesspid{$process_pid}->{HIGH_PCPU_REFILLS};
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index a9c87ad..5f889f1 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -147,7 +147,7 @@ DEFINE_EVENT(kmem_free, kmem_cache_free,
TP_ARGS(call_site, ptr)
);

-TRACE_EVENT(mm_page_free_direct,
+TRACE_EVENT(mm_page_free,

TP_PROTO(struct page *page, unsigned int order),

@@ -169,7 +169,7 @@ TRACE_EVENT(mm_page_free_direct,
__entry->order)
);

-TRACE_EVENT(mm_pagevec_free,
+TRACE_EVENT(mm_page_free_batched,

TP_PROTO(struct page *page, int cold),

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0562d85..2104e23 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -654,7 +654,7 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
int i;
int bad = 0;

- trace_mm_page_free_direct(page, order);
+ trace_mm_page_free(page, order);
kmemcheck_free_shadow(page, order);

if (PageAnon(page))
@@ -1218,7 +1218,7 @@ void free_hot_cold_page_list(struct list_head *list, int cold)
struct page *page, *next;

list_for_each_entry_safe(page, next, list, lru) {
- trace_mm_pagevec_free(page, cold);
+ trace_mm_page_free_batched(page, cold);
free_hot_cold_page(page, cold);

[Konstantin Khlebnikov]

This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.

It frees pages directly from the list without temporary page-vector.

It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.

bloat-o-meter:

add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
function old new delta
free_hot_cold_page_list - 264 +264
get_page_from_freelist 2129 2132 +3
__pagevec_free 243 239 -4
split_free_page 380 373 -7
release_pages 606 510 -96
free_page_list 188 - -188

v2: Remove list reinititialization.
v3: Always free pages in reverse order.
The most recently added struct page, the most likely to be hot.

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---
include/linux/gfp.h | 1 +

mm/page_alloc.c | 13 +++++++++++++

mm/swap.c | 14 +++-----------
mm/vmscan.c | 20 +-------------------

4 files changed, 18 insertions(+), 30 deletions(-)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 3a76faf..6562958 100644

--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -358,6 +358,7 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
extern void __free_pages(struct page *page, unsigned int order);
extern void free_pages(unsigned long addr, unsigned int order);
extern void free_hot_cold_page(struct page *page, int cold);
+extern void free_hot_cold_page_list(struct list_head *list, int cold);

#define __free_page(page) __free_pages((page), 0)
#define free_page(addr) free_pages((addr), 0)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c

index 9dd443d..5093114 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c

@@ -1211,6 +1211,19 @@ out:
}

/*

+ * Free a list of 0-order pages
+ */

+void free_hot_cold_page_list(struct list_head *list, int cold)
+{
+ struct page *page, *next;
+
+ list_for_each_entry_safe(page, next, list, lru) {
+ trace_mm_pagevec_free(page, cold);
+ free_hot_cold_page(page, cold);
+ }
+}
+

+/*

* split_page takes a non-compound higher-order page, and splits it into
* n (1<<order) sub-pages: page[0..n]

* Each sub-page must be freed individually.

diff --git a/mm/swap.c b/mm/swap.c
index a91caf7..67a09a6 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -585,11 +585,10 @@ int lru_add_drain_all(void)

void release_pages(struct page **pages, int nr, int cold)
{
int i;
- struct pagevec pages_to_free;
+ LIST_HEAD(pages_to_free);
struct zone *zone = NULL;
unsigned long uninitialized_var(flags);

- pagevec_init(&pages_to_free, cold);
for (i = 0; i < nr; i++) {
struct page *page = pages[i];

@@ -620,19 +619,12 @@ void release_pages(struct page **pages, int nr, int cold)

del_page_from_lru(zone, page);
}

- if (!pagevec_add(&pages_to_free, page)) {
- if (zone) {
- spin_unlock_irqrestore(&zone->lru_lock, flags);
- zone = NULL;
- }
- __pagevec_free(&pages_to_free);
- pagevec_reinit(&pages_to_free);
- }

+ list_add(&page->lru, &pages_to_free);

}
if (zone)
spin_unlock_irqrestore(&zone->lru_lock, flags);

- pagevec_free(&pages_to_free);
+ free_hot_cold_page_list(&pages_to_free, cold);
}
EXPORT_SYMBOL(release_pages);

diff --git a/mm/vmscan.c b/mm/vmscan.c

index a1893c0..f4be53d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -728,24 +728,6 @@ static enum page_references page_check_references(struct page *page,

return PAGEREF_RECLAIM;
}

-static noinline_for_stack void free_page_list(struct list_head *free_pages)
-{
- struct pagevec freed_pvec;
- struct page *page, *tmp;
-
- pagevec_init(&freed_pvec, 1);
-
- list_for_each_entry_safe(page, tmp, free_pages, lru) {
- list_del(&page->lru);
- if (!pagevec_add(&freed_pvec, page)) {
- __pagevec_free(&freed_pvec);
- pagevec_reinit(&freed_pvec);
- }
- }
-
- pagevec_free(&freed_pvec);
-}
-
/*
* shrink_page_list() returns the number of reclaimed pages
*/

@@ -1009,7 +991,7 @@ keep_lumpy:

[Konstantin Khlebnikov]

Rename page-free mm tracepoints.

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---

Documentation/trace/tracepoint-analysis.txt | 40 ++++++++++++++-------------
tools/perf/Documentation/examples.txt | 34 +++++++++++------------
2 files changed, 37 insertions(+), 37 deletions(-)

diff --git a/Documentation/trace/tracepoint-analysis.txt b/Documentation/trace/tracepoint-analysis.txt
index 87bee3c..058cc6c 100644
--- a/Documentation/trace/tracepoint-analysis.txt
+++ b/Documentation/trace/tracepoint-analysis.txt
@@ -93,14 +93,14 @@ By specifying the -a switch and analysing sleep, the system-wide events
for a duration of time can be examined.

$ perf stat -a \
- -e kmem:mm_page_alloc -e kmem:mm_page_free_direct \
- -e kmem:mm_pagevec_free \
+ -e kmem:mm_page_alloc -e kmem:mm_page_free \
+ -e kmem:mm_page_free_batched \
sleep 10
Performance counter stats for 'sleep 10':

9630 kmem:mm_page_alloc
- 2143 kmem:mm_page_free_direct
- 7424 kmem:mm_pagevec_free
+ 2143 kmem:mm_page_free
+ 7424 kmem:mm_page_free_batched

10.002577764 seconds time elapsed

@@ -119,15 +119,15 @@ basis using set_ftrace_pid.
Events can be activated and tracked for the duration of a process on a local
basis using PCL such as follows.

- $ perf stat -e kmem:mm_page_alloc -e kmem:mm_page_free_direct \
- -e kmem:mm_pagevec_free ./hackbench 10
+ $ perf stat -e kmem:mm_page_alloc -e kmem:mm_page_free \
+ -e kmem:mm_page_free_batched ./hackbench 10
Time: 0.909

Performance counter stats for './hackbench 10':

17803 kmem:mm_page_alloc
- 12398 kmem:mm_page_free_direct
- 4827 kmem:mm_pagevec_free
+ 12398 kmem:mm_page_free
+ 4827 kmem:mm_page_free_batched

0.973913387 seconds time elapsed

@@ -146,8 +146,8 @@ to know what the standard deviation is. By and large, this is left to the
performance analyst to do it by hand. In the event that the discrete event
occurrences are useful to the performance analyst, then perf can be used.

- $ perf stat --repeat 5 -e kmem:mm_page_alloc -e kmem:mm_page_free_direct
- -e kmem:mm_pagevec_free ./hackbench 10
+ $ perf stat --repeat 5 -e kmem:mm_page_alloc -e kmem:mm_page_free
+ -e kmem:mm_page_free_batched ./hackbench 10
Time: 0.890
Time: 0.895
Time: 0.915
@@ -157,8 +157,8 @@ occurrences are useful to the performance analyst, then perf can be used.
Performance counter stats for './hackbench 10' (5 runs):

16630 kmem:mm_page_alloc ( +- 3.542% )
- 11486 kmem:mm_page_free_direct ( +- 4.771% )
- 4730 kmem:mm_pagevec_free ( +- 2.325% )
+ 11486 kmem:mm_page_free ( +- 4.771% )
+ 4730 kmem:mm_page_free_batched ( +- 2.325% )

0.982653002 seconds time elapsed ( +- 1.448% )

@@ -168,15 +168,15 @@ aggregation of discrete events, then a script would need to be developed.
Using --repeat, it is also possible to view how events are fluctuating over
time on a system-wide basis using -a and sleep.

- $ perf stat -e kmem:mm_page_alloc -e kmem:mm_page_free_direct \
- -e kmem:mm_pagevec_free \
+ $ perf stat -e kmem:mm_page_alloc -e kmem:mm_page_free \
+ -e kmem:mm_page_free_batched \
-a --repeat 10 \
sleep 1
Performance counter stats for 'sleep 1' (10 runs):

1066 kmem:mm_page_alloc ( +- 26.148% )
- 182 kmem:mm_page_free_direct ( +- 5.464% )
- 890 kmem:mm_pagevec_free ( +- 30.079% )
+ 182 kmem:mm_page_free ( +- 5.464% )
+ 890 kmem:mm_page_free_batched ( +- 30.079% )

1.002251757 seconds time elapsed ( +- 0.005% )

@@ -220,8 +220,8 @@ were generating events within the kernel. To begin this sort of analysis, the
data must be recorded. At the time of writing, this required root:

$ perf record -c 1 \
- -e kmem:mm_page_alloc -e kmem:mm_page_free_direct \
- -e kmem:mm_pagevec_free \
+ -e kmem:mm_page_alloc -e kmem:mm_page_free \
+ -e kmem:mm_page_free_batched \
./hackbench 10
Time: 0.894
[ perf record: Captured and wrote 0.733 MB perf.data (~32010 samples) ]
@@ -260,8 +260,8 @@ noticed that X was generating an insane amount of page allocations so let's look
at it:

$ perf record -c 1 -f \
- -e kmem:mm_page_alloc -e kmem:mm_page_free_direct \
- -e kmem:mm_pagevec_free \
+ -e kmem:mm_page_alloc -e kmem:mm_page_free \
+ -e kmem:mm_page_free_batched \
-p `pidof X`

This was interrupted after a few seconds and
diff --git a/tools/perf/Documentation/examples.txt b/tools/perf/Documentation/examples.txt
index 8eb6c48..77f9527 100644
--- a/tools/perf/Documentation/examples.txt
+++ b/tools/perf/Documentation/examples.txt
@@ -17,8 +17,8 @@ titan:~> perf list
kmem:kmem_cache_alloc_node [Tracepoint event]
kmem:kfree [Tracepoint event]
kmem:kmem_cache_free [Tracepoint event]
- kmem:mm_page_free_direct [Tracepoint event]
- kmem:mm_pagevec_free [Tracepoint event]
+ kmem:mm_page_free [Tracepoint event]
+ kmem:mm_page_free_batched [Tracepoint event]
kmem:mm_page_alloc [Tracepoint event]
kmem:mm_page_alloc_zone_locked [Tracepoint event]
kmem:mm_page_pcpu_drain [Tracepoint event]
@@ -29,15 +29,15 @@ measured. For example the page alloc/free properties of a 'hackbench
run' are:

titan:~> perf stat -e kmem:mm_page_pcpu_drain -e kmem:mm_page_alloc
- -e kmem:mm_pagevec_free -e kmem:mm_page_free_direct ./hackbench 10
+ -e kmem:mm_page_free_batched -e kmem:mm_page_free ./hackbench 10
Time: 0.575

Performance counter stats for './hackbench 10':

13857 kmem:mm_page_pcpu_drain
27576 kmem:mm_page_alloc
- 6025 kmem:mm_pagevec_free
- 20934 kmem:mm_page_free_direct
+ 6025 kmem:mm_page_free_batched
+ 20934 kmem:mm_page_free

0.613972165 seconds time elapsed

@@ -45,8 +45,8 @@ You can observe the statistical properties as well, by using the
'repeat the workload N times' feature of perf stat:

titan:~> perf stat --repeat 5 -e kmem:mm_page_pcpu_drain -e
- kmem:mm_page_alloc -e kmem:mm_pagevec_free -e
- kmem:mm_page_free_direct ./hackbench 10
+ kmem:mm_page_alloc -e kmem:mm_page_free_batched -e
+ kmem:mm_page_free ./hackbench 10
Time: 0.627
Time: 0.644
Time: 0.564
@@ -57,8 +57,8 @@ You can observe the statistical properties as well, by using the

12920 kmem:mm_page_pcpu_drain ( +- 3.359% )
25035 kmem:mm_page_alloc ( +- 3.783% )
- 6104 kmem:mm_pagevec_free ( +- 0.934% )
- 18376 kmem:mm_page_free_direct ( +- 4.941% )
+ 6104 kmem:mm_page_free_batched ( +- 0.934% )
+ 18376 kmem:mm_page_free ( +- 4.941% )

0.643954516 seconds time elapsed ( +- 2.363% )

@@ -158,15 +158,15 @@ Or you can observe the whole system's page allocations for 10
seconds:

titan:~/git> perf stat -a -e kmem:mm_page_pcpu_drain -e
-kmem:mm_page_alloc -e kmem:mm_pagevec_free -e
-kmem:mm_page_free_direct sleep 10
+kmem:mm_page_alloc -e kmem:mm_page_free_batched -e
+kmem:mm_page_free sleep 10

Performance counter stats for 'sleep 10':

171585 kmem:mm_page_pcpu_drain
322114 kmem:mm_page_alloc
- 73623 kmem:mm_pagevec_free
- 254115 kmem:mm_page_free_direct
+ 73623 kmem:mm_page_free_batched
+ 254115 kmem:mm_page_free

10.000591410 seconds time elapsed

@@ -174,15 +174,15 @@ Or observe how fluctuating the page allocations are, via statistical
analysis done over ten 1-second intervals:

titan:~/git> perf stat --repeat 10 -a -e kmem:mm_page_pcpu_drain -e
- kmem:mm_page_alloc -e kmem:mm_pagevec_free -e
- kmem:mm_page_free_direct sleep 1
+ kmem:mm_page_alloc -e kmem:mm_page_free_batched -e
+ kmem:mm_page_free sleep 1

Performance counter stats for 'sleep 1' (10 runs):

17254 kmem:mm_page_pcpu_drain ( +- 3.709% )
34394 kmem:mm_page_alloc ( +- 4.617% )
- 7509 kmem:mm_pagevec_free ( +- 4.820% )
- 25653 kmem:mm_page_free_direct ( +- 3.672% )
+ 7509 kmem:mm_page_free_batched ( +- 4.820% )
+ 25653 kmem:mm_page_free ( +- 3.672% )

1.058135029 seconds time elapsed ( +- 3.089% )

[Konstantin Khlebnikov]

It not exported and now nobody use it.

Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>
---

include/linux/pagevec.h | 7 -------
mm/page_alloc.c | 10 ----------
2 files changed, 0 insertions(+), 17 deletions(-)

diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index bab82f4..ed17024 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -21,7 +21,6 @@ struct pagevec {
};

void __pagevec_release(struct pagevec *pvec);
-void __pagevec_free(struct pagevec *pvec);
void ____pagevec_lru_add(struct pagevec *pvec, enum lru_list lru);
void pagevec_strip(struct pagevec *pvec);
unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
@@ -67,12 +66,6 @@ static inline void pagevec_release(struct pagevec *pvec)
__pagevec_release(pvec);
}

-static inline void pagevec_free(struct pagevec *pvec)
-{
- if (pagevec_count(pvec))
- __pagevec_free(pvec);
-}
-
static inline void __pagevec_lru_add_anon(struct pagevec *pvec)
{
____pagevec_lru_add(pvec, LRU_INACTIVE_ANON);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5093114..0562d85 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2341,16 +2341,6 @@ unsigned long get_zeroed_page(gfp_t gfp_mask)
}
EXPORT_SYMBOL(get_zeroed_page);

-void __pagevec_free(struct pagevec *pvec)
-{
- int i = pagevec_count(pvec);
-
- while (--i >= 0) {
- trace_mm_pagevec_free(pvec->pages[i], pvec->cold);
- free_hot_cold_page(pvec->pages[i], pvec->cold);
- }
-}
-

void __free_pages(struct page *page, unsigned int order)

{
if (put_page_testzero(page)) {

[Minchan Kim]

On Fri, Nov 11, 2011 at 10:40 PM, Konstantin Khlebnikov
<khleb...@openvz.org> wrote:
> Rename mm_page_free_direct into mm_page_free
> and mm_pagevec_free into mm_page_free_batched
>
> Since v2.6.33-5426-gc475dab kernel trigger mm_page_free_direct for all freed pages,
> not only for directly freed. So, let's name it properly.
> For pages freed via page-list we also trigger mm_page_free_batched event.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

Reviewed-by: Minchan Kim <minch...@gmail.com>

1+ for clear naming but I am not quite sure event name change is
always OK for compatibility of old perf.
At least, we will need to Cced Ingo, I think.

--
Kind regards,
Minchan Kim

[Minchan Kim]

On Fri, Nov 11, 2011 at 10:40 PM, Konstantin Khlebnikov
<khleb...@openvz.org> wrote:

> It not exported and now nobody use it.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

Reviewed-by: Minchan Kim <minch...@gmail.com>

--
Kind regards,
Minchan Kim

[Minchan Kim]

On Fri, Nov 11, 2011 at 10:39 PM, Konstantin Khlebnikov
<khleb...@openvz.org> wrote:
> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> It frees pages directly from the list without temporary page-vector.
> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
>
> bloat-o-meter:
>
> add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
> function                                     old     new   delta
> free_hot_cold_page_list                        -     264    +264
> get_page_from_freelist                      2129    2132      +3
> __pagevec_free                               243     239      -4
> split_free_page                              380     373      -7
> release_pages                                606     510     -96
> free_page_list                               188       -    -188
>
> v2: Remove list reinititialization.
> v3: Always free pages in reverse order.
>    The most recently added struct page, the most likely to be hot.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

[Minchan Kim]

On Fri, Nov 11, 2011 at 10:40 PM, Konstantin Khlebnikov
<khleb...@openvz.org> wrote:
> Rename page-free mm tracepoints.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

Reviewed-by: Minchan Kim <minch...@gmail.com>

--
Kind regards,
Minchan Kim

[Hugh Dickins]

On Fri, 11 Nov 2011, Konstantin Khlebnikov wrote:

> This patch adds helper free_hot_cold_page_list() to free list of 0-order pages.
> It frees pages directly from the list without temporary page-vector.
> It also calls trace_mm_pagevec_free() to simulate pagevec_free() behaviour.
>
> bloat-o-meter:
>
> add/remove: 1/1 grow/shrink: 1/3 up/down: 267/-295 (-28)
> function old new delta
> free_hot_cold_page_list - 264 +264
> get_page_from_freelist 2129 2132 +3
> __pagevec_free 243 239 -4
> split_free_page 380 373 -7
> release_pages 606 510 -96
> free_page_list 188 - -188
>
> v2: Remove list reinititialization.
> v3: Always free pages in reverse order.
> The most recently added struct page, the most likely to be hot.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

Acked-by: Hugh Dickins <hu...@google.com>

[Hugh Dickins]

On Fri, 11 Nov 2011, Konstantin Khlebnikov wrote:

> It not exported and now nobody use it.
>
> Signed-off-by: Konstantin Khlebnikov <khleb...@openvz.org>

Acked-by: Hugh Dickins <hu...@google.com>