How to limit hugepage allocation

1,748 views
Skip to first unread message

Maik Pfeil

unread,
Nov 14, 2017, 11:41:34 AM11/14/17
to TRex Traffic Generator
Hello,

I reserved 24 x 1G hugepages on kernel bootup for some DPDK applications. Once I start TRex I can see that it allocate all remaining hugepages on all NUMA nodes. For eg:

user@host:~$ sudo grep huge /proc/*/numa_maps
<snip>
/proc/17459/numa_maps:7f73c0000000 default file=/mnt/huge/rtemap_19 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7400000000 default file=/mnt/huge/rtemap_18 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7440000000 default file=/mnt/huge/rtemap_17 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7480000000 default file=/mnt/huge/rtemap_16 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f74c0000000 default file=/mnt/huge/rtemap_15 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7500000000 default file=/mnt/huge/rtemap_14 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7540000000 default file=/mnt/huge/rtemap_13 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7580000000 default file=/mnt/huge/rtemap_12 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f75c0000000 default file=/mnt/huge/rtemap_11 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7600000000 default file=/mnt/huge/rtemap_10 huge dirty=1 N1=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7680000000 default file=/mnt/huge/rtemap_9 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f76c0000000 default file=/mnt/huge/rtemap_8 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7700000000 default file=/mnt/huge/rtemap_7 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7740000000 default file=/mnt/huge/rtemap_6 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7780000000 default file=/mnt/huge/rtemap_5 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f77c0000000 default file=/mnt/huge/rtemap_4 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7800000000 default file=/mnt/huge/rtemap_3 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7840000000 default file=/mnt/huge/rtemap_2 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f7880000000 default file=/mnt/huge/rtemap_1 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex
/proc/17459/numa_maps:7f78c0000000 default file=/mnt/huge/rtemap_0 huge dirty=1 N0=1 kernelpagesize_kB=1048576   <<< TRex

user@host:~$ ps uxaw | grep 17459
root     17459  5.6  0.0 21780464 11932 pts/1  Sl+  17:35   0:09 ./_t-rex-64 -i --iom 1 -c 2

In this example TRex tookover 20 x 1G of the hugepages, 4 x 1G left used by other DPDK applications. 

Is it possible to limit the amount of hugepages/rtemaps that TRex allocate on startup? 

I use v2.28.


Best regards
Maik

hanoh haim

unread,
Nov 14, 2017, 2:26:21 PM11/14/17
to Maik Pfeil, TRex Traffic Generator
Hi 

see here, 
use this in the config file 
       prefix          : trex1 
       limit_memory    : 2048

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_yaml_configuration_file_parameter_of_cfg_option


--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/ee2e3610-00e4-486b-898a-b5cb87f3a509%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Hanoh
Sent from my iPhone

Maik Pfeil

unread,
Nov 14, 2017, 4:45:58 PM11/14/17
to TRex Traffic Generator
Thanks Hanoh, I will try it.

- Maik

Maik Pfeil

unread,
Nov 15, 2017, 2:05:26 AM11/15/17
to TRex Traffic Generator
Hi Hanoh,

I tried it with the prefix/limit_memory option in trex_cfg.yaml:

- port_limit      : 6
  version         : 2
  interfaces      : ["04:00.1","07:00.1","04:00.0","07:00.0","21:00.1","24:00.1"]
  prefix          : trex1
  limit_memory    : 1024
  port_info       :
          - ip         : 60.99.99.1
            default_gw : 60.99.99.2
          - ip         : 60.99.99.5
            default_gw : 60.99.99.6
          - ip         : 60.99.99.9
            default_gw : 60.99.99.10
          - ip         : 60.99.99.13
            default_gw : 60.99.99.14
          - ip         : 60.99.99.17
            default_gw : 60.99.99.18
          - ip         : 60.99.99.21
            default_gw : 60.99.99.22
  platform :
        master_thread_id : 10
        latency_thread_id : 11
        dual_if :
             - socket : 0
               threads : [2,3]
             - socket : 0
               threads : [8,9]
             - socket : 1
               threads : [12,5]

I if start trex with : 

user@host:~/trex/v2.28$ sudo ./t-rex-64 -i --iom 1 -c 1

it comes up and reserve rtemaps per socket:

user@host:~$ sudo grep huge /proc/*/numa_maps
/proc/14741/numa_maps:7f79c0000000 default file=/vppmap_5 huge dirty=1 N1=1 kernelpagesize_kB=1048576
/proc/14741/numa_maps:7f7cc0000000 default file=/vppmap_11 huge dirty=1 N0=1 kernelpagesize_kB=1048576
/proc/14762/numa_maps:7fa7c0000000 default file=/vppmap_5 huge dirty=1 N1=1 kernelpagesize_kB=1048576
/proc/14762/numa_maps:7fab00000000 default file=/vppmap_11 huge dirty=1 N0=1 kernelpagesize_kB=1048576
/proc/20916/numa_maps:7fac00000000 default file=/mnt/huge/trex1map_10 huge dirty=1 N1=1 kernelpagesize_kB=1048576    <<<
/proc/20916/numa_maps:7faec0000000 default file=/mnt/huge/trex1map_0 huge dirty=1 N0=1 kernelpagesize_kB=1048576    <<<

But if I try to start TRex with 2 cores per port assigned :

user@host:~/trex/v2.28$ sudo ./t-rex-64 -i --iom 1 -c 2

I get following error:

RING: Cannot reserve memory
 ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
  Cause: Cannot init nodes mbuf pool nodes-3

At this time I have 20 x 1Gb hugepages not in use on the system and only 4 hugepages for vpp allocated:

user@host:~$ sudo grep huge /proc/*/numa_maps
/proc/14741/numa_maps:7f79c0000000 default file=/vppmap_5 huge dirty=1 N1=1 kernelpagesize_kB=1048576
/proc/14741/numa_maps:7f7cc0000000 default file=/vppmap_11 huge dirty=1 N0=1 kernelpagesize_kB=1048576
/proc/14762/numa_maps:7fa7c0000000 default file=/vppmap_5 huge dirty=1 N1=1 kernelpagesize_kB=1048576
/proc/14762/numa_maps:7fab00000000 default file=/vppmap_11 huge dirty=1 N0=1 kernelpagesize_kB=1048576

user@host:~$ sudo hugeadm --pool-list
      Size  Minimum  Current  Maximum  Default
1073741824       24       24       24        *

Do you have an idea?

Could you change documentation that "limit_memory" option need always "prefix" option to work, tried it first without prefix with no change to dpdk.


- Maik

hanoh haim

unread,
Nov 15, 2017, 2:16:45 AM11/15/17
to Maik Pfeil, TRex Traffic Generator
Two way to solve this 
1) from CLI add something like that 
 --mbuf-factor 0.2 
tune the factor to your need. This will reduce the amount of mbuf memory 
2) add more memory 

 limit_memory    : 1024

==>

try  1536 or 2048


thanks,
Hanoh
--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

Maik Pfeil

unread,
Nov 20, 2017, 2:41:40 PM11/20/17
to hanoh haim, TRex Traffic Generator
Hi Hanoh,

limit_memory: 2048 works fine! Thank you!


- Maik
Reply all
Reply to author
Forward
0 new messages