transparent_hugepages

[Dave Howorth]

I run openSUSE Leap 15.0. It seems that by default:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

and consequently in my redis-server log I see:

1488:M 16 Mar 09:39:56.590 # WARNING you have Transparent Huge Pages
(THP) support enabled in your kernel. This will create latency and
memory usage issues with Redis. To fix this issue run the command 'echo
never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add
it to your /etc/rc.local in order to retain the setting after a reboot.
Redis must be restarted after THP is disabled.

After looking up what THP are (I know nothing about them) it seemed to
me that it ought to be safe to use a setting of madvise rather than
never. So:

# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
# systemctl restart redis@default

and now in my log I have (I also switched logging from a file to the
journal):
Mar 17 11:23:44 acer-suse redis-server[17347]: 17347:M 17 Mar
11:23:44.748 # WARNING you have Transparent Huge Pages (THP) support
enabled in your kernel. This will create latency and memory usage
issues with Redis. ...

So is my conclusion that madvise is safe wrong? Or is redis being
unduly paranoid in its warning? And why is it suggesting using rc.local
on a systemd installation?

[hva...@gmail.com]

My reading of the documentation on THP shows this:

"always" means that an application requesting THP will stall on allocation
failure and directly reclaim pages and compact memory in an effort to
allocate a THP immediately. This may be desirable for virtual machines
that benefit heavily from THP use and are willing to delay the VM start
to utilise them.

"madvise" will enter direct reclaim like "always" but only for regions
that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.

I don't see a difference that helps Redis by setting "madvise", and the doco indicates it's already the default.  Are you running something on the same server that needs THP?

The warning message in Redis was probably written before so many distros switched to Systemd, and hasn't been re-written since.  I'm sure a pull request with a re-wording will be welcomed by the development team.

[Dave Howorth]

On Sun, 17 Mar 2019 20:54:07 -0700 (PDT)
hva...@gmail.com wrote:

> My reading of the documentation on THP shows this:

A link to the doco you are using would be helpful.

> "always" means that an application requesting THP will stall on
> allocation failure and directly reclaim pages and compact memory in
> an effort to allocate a THP immediately. This may be desirable for
> virtual machines that benefit heavily from THP use and are willing to
> delay the VM start to utilise them.

Yup.

> "madvise" will enter direct reclaim like "always" but only for regions
> that are have used madvise(MADV_HUGEPAGE).

Yup.

> This is the default behaviour.

This is where a link to the doco would be useful, since as I said:

> > I run openSUSE Leap 15.0. It seems that by default:
> >
> > $ cat /sys/kernel/mm/transparent_hugepage/enabled
> > [always] madvise never

So on which systems is madvise the default?

> I don't see a difference that helps Redis by setting "madvise",

> the doco indicates it's already the default. Are you running
> something on the same server that needs THP?

It's not a server; it's my desktop machine. I know nothing about THP,
as I said, so I do not know whether anything needs or even benefits
from them. But as I also said, my reading suggested that madvise ought
to benefit redis, since it will presumably then not be subject to THP
and subsequent delays.

> The warning message in Redis was probably written before so many
> distros switched to Systemd, and hasn't been re-written since. I'm
> sure a pull request with a re-wording will be welcomed by the
> development team.

You may be right, but since I know nothing about THP and very little
about redis and am asking for help on here, I doubt I'm the person to
write it.

[hva...@gmail.com]

I did a Google search for [ linux kernel "transparent_hugepage" ]  (without the square brackets).  The first answer was from kernel.org:  https://www.kernel.org/doc/Documentation/vm/transhuge.txt .  The sentence "This is the default behavior" was in that doc.

Apparently the maintainers of your Linux distribution changed the kernel config from the documented default in the code.  That's not unusual, though.

The first paragraph from the doc page I linked says:

Performance critical computing applications dealing with large memory
working sets are already running on top of libhugetlbfs and in turn
hugetlbfs. Transparent Hugepage Support is an alternative means of
using huge pages for the backing of virtual memory with huge pages
that supports the automatic promotion and demotion of page sizes and
without the shortcomings of hugetlbfs.

The Linux kernel has many, many configuration options, and it's easy to start sniffing down the trail of one or two of them, thinking you will reap performance rewards, only to find yourself at the end of a time-consuming journey that improves your daemon's performance by a mere 0.1% with the hardware you have, or with the traffic you'll receive for the first 2 years of operation, or the size of your data set for those first 2 years.  The first 10 words of the THP doc (yellow highlighted by me) indicates it's designed for situations that will almost never be found on a laptop:  (a) performance critical and (b) large amounts of data in memory.

I say "performance critical" won't be found on most desktop or laptop machines because the overwhelming majority of performance hits a daemon like Redis sees on these machines come from the other software components it must share memory and cpu with - the graphical display's windowing system, unexpected bursts of cpu, memory, and disk io usage from compilers or high-level language runtime environments (Java, Ruby, Python, etc.), any of which will completely overshadow any impediments from not using THP.

I would like to suggest that you don't need to go against Redis's advice about THP unless you're handling hundreds of GB of data in memory on a server that's dedicated to Redis, where you've solved the other sources of latency discussed in https://redis.io/topics/latency (which describes the kind of latency you get when THP is enabled), so the primary source of Redis performance trouble will come down to a kernel tuning issue.

[Dave Howorth]

On Tue, 19 Mar 2019 00:45:36 -0700 (PDT)
hva...@gmail.com wrote:

> I did a Google search for [ linux kernel "transparent_hugepage" ]
> (without the square brackets). The first answer was from
> kernel.org:
> https://www.kernel.org/doc/Documentation/vm/transhuge.txt . The
> sentence "This is the default behavior" was in that doc.
>
> Apparently the maintainers of your Linux distribution changed the
> kernel config from the documented default in the code. That's not
> unusual, though.

Thanks. I hadn't seen that page. I didn't use 'kernel' when searching
so perhaps that's why. BTW, the text doesn't contain the sentence you
'quoted'. The text contains the word 'behaviour' rather than 'behavior'.

The madvise default you mention is in repect of the defrag option, not
the enabled option, and indeed the default on my openSUSE system of the
defrag option is 'madvise'. But it's the enabled option that I'm
interested in and a default for that doesn't appear to be stated.

> The first paragraph from the doc page I linked says:
>
> Performance critical computing applications dealing with large memory
> working sets are already running on top of libhugetlbfs and in turn
> hugetlbfs. Transparent Hugepage Support is an alternative means of
> using huge pages for the backing of virtual memory with huge pages
> that supports the automatic promotion and demotion of page sizes and
> without the shortcomings of hugetlbfs.
>
> The Linux kernel has many, many configuration options, and it's easy
> to start sniffing down the trail of one or two of them, thinking you
> will reap performance rewards, only to find yourself at the end of a
> time-consuming journey that improves your daemon's performance by a
> mere 0.1% with the hardware you have, or with the traffic you'll
> receive for the first 2 years of operation, or the size of your data
> set for those first 2 years. The first 10 words of the THP doc
> (yellow highlighted by me)

Sorry, I read plain text email. But I can count :)

> indicates it's designed for situations
> that will almost never be found on a laptop: (a) performance
> critical and (b) large amounts of data in memory.

True. Fortunately (?) I don't have any laptops.

> I say "performance critical" won't be found on most desktop or laptop
> machines because the overwhelming majority of performance hits a
> daemon like Redis sees on these machines come from the other software
> components it must share memory and cpu with - the graphical
> display's windowing system, unexpected bursts of cpu, memory, and
> disk io usage from compilers or high-level language runtime
> environments (Java, Ruby, Python, etc.), any of which will completely
> overshadow any impediments from not using THP.
>
> I would like to suggest that you don't need to go against Redis's
> advice about THP unless you're handling hundreds of GB of data in
> memory on a server that's dedicated to Redis, where you've solved the
> other sources of latency discussed in https://redis.io/topics/latency
> (which describes the kind of latency you get when THP is enabled), so
> the primary source of Redis performance trouble will come down to a
> kernel tuning issue.

Well, I tend to agree with your opinion, but equally I tend to trust
openSUSE's settings and don't like arbitrary changes - as you say
chasing rabbits down a hole can be pointless, and I also don't
understand why redis' problem wouldn't be solved by a madvise setting.

Maybe I'll bug-report both redis and openSUSE and see who comes up with
the best-reasoned explanation. But I'd rather avoid bothering the devs
if anybody else actually knows the answers.

Thanks for investigating this with me.

[Anthony LaTorre]

For what it's worth, I have the exact same settings on a server I
manage (CentOS 6) and have been running Redis for 5 years 24/7 with ~
10k writes/s using on average about 2-3 GB of memory and have never
run into any latency or memory issues.

This is just one data point and it may have an impact on different
setups, but I haven't had any issues.

Tony

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[re...@howorth.org.uk]

> For what it's worth, I have the exact same settings on a server

Thanks for this and I'm probably being dumb but which settings do you
mean? always, never or madvise?

[Anthony LaTorre]

I meant I had the same THP settings as you originally reported:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

Tony

[re...@howorth.org.uk]

> I meant I had the same THP settings as you originally reported:
>
> $ cat /sys/kernel/mm/transparent_hugepage/enabled
> [always] madvise never

OK, thanks.