Hugepages?

[Matt Welch]

Hi GENI Users,

This started out as a question, but has turned into an informative post :)

I'm doing an experiment with DPDK (Data Plane Dev Kit) which can greatly accelerate network processing when used with memory hugepages. 
Hugepages can actually improve performance on many types of workloads, especially those that are memory intensive and will swap pages frequently when they are 4k. 
They can be configured in a few different sizes, depending on the architecture.  Intel systems can usually use up to 1GB superpages. 

I've modified my kernel configuration to support hugepages and added the correct kernel parameters to support them: default_hugepagesz=2M hugepagesz=2M hugepages=4096

So, it seems that, if I configure the kernel properly with kernel parameters, I can get the number of hugepages that I need:
mattwel@node-0:~$ cat /proc/meminfo | grep Huge
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

My concern was that I don't appear to be able to change them (in the manner I was used to) in a running system and get the following error:
mattwel@node-0:~$ sudo echo 2048 > /proc/sys/vm/nr_hugepages
-bash: /proc/sys/vm/nr_hugepages: Permission denied

My solution is this: use sysctl!
mattwel@node-0:~$ sudo sysctl -w vm.nr_hugepages=2048
vm.nr_hugepages = 2048
mattwel@node-0:~$ cat /proc/meminfo | grep Huge
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

So sysctl allowed me to change hugepages, but writing to the procfs directly doesn't - no problem! 

I hope that my hugepage adventure will help someone improve their application performance!

Cheers,
Matt Welch

[Nicholas Bastin]

On Wed, Aug 5, 2015 at 3:12 PM, Matt Welch <matt...@gmail.com> wrote:

mattwel@node-0:~$ sudo echo 2048 > /proc/sys/vm/nr_hugepages
-bash: /proc/sys/vm/nr_hugepages: Permission denied

This is because you are running 'echo' as sudo, but the output redirection ('>') runs as your user (as it is executed by the shell that you own), and you do not have permission to write into proc.

When writing into proc using sudo you need to invoke a new shell that runs both your command and the output redirection:

sudo sh -c 'echo 2048 > /proc/sys/vm/nr_hugepages'

or

echo "2048" | sudo tee /proc/sys/vm/nr_hugepages

Also most geni resources don't have NICs that support DPDK, so you might encounter further problems depending on the resources you are using.

--

Nick 

[Matt Welch]

Thanks Nick,

You're absolutely right there - I didn't even think about the redirection. 

Re: the NIC, I'm using raw-pc in my topology and they thankfully have the Intel I350 in them.  According to the dpdk.org docs on supported nics, it should work so I'm hopeful!!

Thanks for the heads-up though. 
( I really love this community BTW, you all are great! )

Cheers,
Matt

[Nicholas Bastin]

On Wed, Aug 5, 2015 at 3:35 PM, Matt Welch <matt...@gmail.com> wrote:

Re: the NIC, I'm using raw-pc in my topology and they thankfully have the Intel I350 in them.  According to the dpdk.org docs on supported nics, it should work so I'm hopeful!!

Yep, modern Intel NICs are fine.  A lot of GENI resources have non-Intel NICs (mlx3, chelsio, netXtreme, etc.) for which there are no DPDK drivers (yet, at least). 

--

Nick